Because of this we created Bankscrap[1], a Ruby gem to access multiple banks. We basically find the APIs that the Banks are using for their mobile apps and expose them through a common Ruby library with an unified data model.
Each bank has an open source adapter (this is different to Teller) and we encourage the community to help us building more adapters. So far we got adapters for 4 major banks in Spain, and 3 more are a work in progress.
Whether PSD2 is going to happen or not, we believe public APIs for banks will happen (even if banks don't like it).
IMHO banks arte not scared because of security concerns: they all have APIs in production already, they are just not documented. Their main concerns is basically how APIs used by third party services could affect their businesses.
So who bears the legal responsibility? Is it the business which uses your gems or the customer? You're effectively breaking most banks' terms of service, at least in the UK. It's great that you've solved the tech challenge, but that's not the biggest one to solve.
From our understanding what we do is legal in the EU, according to the European Union Computers Programs directive 2009:
> The person having a right to use a copy of a computer
program shall be entitled, without the authorisation of the rightholder, to observe, study or test the functioning of the program in order to determine the ideas and principles which underline any element of the program if he does so while performing any of the acts of loading, displaying, running, transmitting or storing the program which he is entitled to do.
There is in fact a EU proceeding against MathWorks for preventing a competitor to reverse-engineering their product to achieve interoperability [2]
In any case, we currently offer no guarantee to our users and we encourage them to use our open source libraries at their own risk.
I'm not sure you've addressed the concern. Sure, reverse-engineering an interface is legal. However, use of these APIs by third-party, non-vetted apps or libraries potentially incurs other legal risks.
I suppose you've somewhat addressed it with your final comment: "...at their own risk." I'm not risking access to all my money on using your library to access my bank account. If you say the risk is mine, and, since I'm using your library rather than an approved binary, the bank can show that the risk is mine, when someone finds and exploits a weakness in your library to take my money, I'm screwed.
> when someone finds and exploits a weakness in your library to take my money, I'm screwed.
I'm not sure what kind of exploit or weakness you are thinking about. But the risk of using Bankscrap's gem from your computer is similar to using your bank's mobile app. Bankscrap itself doesn't store any data nor send your credentials to any place other than your bank's API (the same that your mobile app does).
In other words, if you use Bankscrap's gem to build a web app to fetch and store your bank transactions (or someone else's), and your webapp or server gets hacked, obviously the responsibility is yours.
If you had your own API exposed to the internet and someone made an HTTP call that "triggers a bug" in your server "and takes the whole system offline", would you sue the guy who made the HTTP call or would you assume it was a bug in your server / app?
And we're not talking about HTTP calls with SQL injection or things like that. We're talking about identical HTTP calls to the ones that the banks' mobile apps do. Same headers, same body.
If they're identical to the official app, they likely won't trigger an unexpected case. If they're not identical (maybe your open source lib has a bug, or the API changed) I could see lawyers trying to sell it as a deliberate hacking attempt to a court.
The open source library simply helps a programmer put together HTTP requests. If an API crashes because it receives an unexpected HTTP request its certainly not the requesters fault.
At Delft University of Technology we're working on this also for about 2 years. Currently 8 (big) banks reverse-engineered, the smartphone way (no scraping).
Besides universities there are not many non-profits that will have the legal and financial means to publish an open source lib. The legal landscape for managing a complete library with numerous banks is complex. As we see in this discussion thread, there are some fragmented solutions with limited coverage.
That is reasonable technical approach to the problem, but not how the courts work. On the other hand in court also should ask and answer the question of whether you triggered such behavior intentionally or not.
Also, if it's their public API used by their mobile/web apps, it would likely be crashing all the time anyway from normal user interaction (if some OSS were able to crash it via HTTP).
Nah, you can totally walk a tightrope where one's app is never crashes the API, but only because it has no input fields that allow you to try to transfer currency in the amount of "spaghetti" from one account to another.
You make it sound silly. But the "unexpected HTTP request" may in fact be a SYN flood (yes, that has happened), or looping some non-trivial request for an ad-hoc DoS. That is indeed the requesters "fault", even if there is clearly no intent and hopefully no legal ramifications.
Sorry, but I still don't understand what you mean here. There could be a bug in the library in the same manner that there could be bugs in any other open source library. Could those bugs become a security issue for your app? Yes, they could. Despite of this we all use open source software, right?
Let's say you were using Active Merchant[1], would you sue Shopify if someone hacked your app to process irregular payments through your Bank's payment gateway (accessed by Active Merchant [2])?
If someone hacks your business checking account, you have 24 hours to detect and report the unauthorized transfer. After that, the UCC doesn't require the bank to give you your money back.
And yet every business has a checking account, and many aren't that diligent about checking it every 24 hours.
Bankscrap, despite of its name, tries to avoid web scraping in favor of XML/JSON apis provided by the banks. Right now all our working adapters are API-based, not web scrapers.
It seems to a be very recurring discussion in the comments on each LinuxFR release post.
You can add that some modules have the same kind of name (BNporc which would translate to BNswine for BNP Paribas[0]). Or some applications: QHandjoob[1], QFlatBoob[2] or QHavedate[3] where the icon is a stickman with an erect penis getting ready to have sex doggy style[4].
They used to have a nazi svatiska on the Caisse d'Épargne module logo[5]. At least they changed that one.
Really, how anyone could use that professionally is beyond me.
Does it break functionality if you don't pretend to be an Android phone?
Adding the library version in their would be helpful for debugging too. Ideally it'd be possible to override at the app level so that it correctly shows up in their logs.
I used to work in banking security and I can understand the bank's concerns. If they allow a 3rd party access to customer data and that customer data is subsequently leaked and used for fraud, who's liable?
If a law could codify an acceptable answer to that problem, I think a lot of the security/regulatory problems could go away. Banks might still try to stop the process for competitive reasons, but they might not be able to lean on the crutch of "security" to do so.
In the UK this problem isn't new, there were aggregator services 15 years ago that screen-scraped banking data to provide customers with a consolidated view of their finances and they required customer's to provide their credentials for them to do that. Banks understandably weren't too pleased about the idea of 3rd parties having those credentials.
In Norway it is quite clear who is liable, the 3rd party.
Anyone that holds personal data needs to acquire a license to do so from Datatilsynet (data protection authority) and they can only use it for the purpose specified/granted in the license.
Most of the banks here just provide downloadable csv/excel files after logging in with BankID (national electronic ID thing, https://www.bankid.no/en/company/).
If you upload that file to a 3rd party, the 3rd party still needs a license etc to keep/process it and is liable.
I'd argue the same would be true in the UK too - under the Data Protection Act, any company who holds personal information is required to register with a government authority [0], and the act makes it fairly clear what responsibilities and actions need to be taken regarding the handling of data.
It's a little bit more nuanced than that. The Act places an obligation on data controllers to register and to comply with the various obligations under the Act. This excludes data processors (with controllers then making sure they place relevant obligations on controllers contractually). Potentially a service provider might argue they're a data processor although if they're taking decisions over how the data is used they could easily fall within the data controller category. Under current law, it's the data controller who would be liable for any loss of data (although they would normally look to cover off liability contractually with the processor).
In any event, liability really depends on who is taking the decisions over the use of the data. If the third party just takes a feed of the data from the bank then takes various decisions over its use, they could well be a data controller. If on the other hand a bank contracts with a third party to provide an account aggregation service to its customers and they are passing data for this very specific purpose then the service provider could well just be a processor.
The above is also going to change with the new General Data Protection Regulation coming into force next year...
>Anyone that holds personal data needs to acquire a license to do so from Datatilsynet
That sounds ridiculous. Are YCombinator executives at risk of being fined/jailed in Norway since they presumably did not get a license for the Hacker News users table?
Would a Norwegian teenager land in legal trouble for running an unlicensed phpBB forum? Do SaaS programmers need code review from the government on their pull requests? When Amazon or Netflix trains a new recommendations model?
the reason is simple - IT is considered just annoying cost center that prevents rest of the bank to reach the stars... or something. budgeting reflects this accordingly
Actually, in EU almost anyone with a little bit of money could start a financial institute that would be able to lend money to people using real estate as security, backed by the central banks.
The more complicated bank licence is needed to lend money from people.
I think this has historically been the main reason. The computers where introduces to reduce the operational costs, and that mentality has stuck.
But I think most banks have realized that they need to do something, lest they will end up the non-significant bill for the infrastructure but a dwindling list of services that they can make money on.
Regarding third party screen-scrapers, from what I can see there are challenges both in the legal framework and technical issues like security and performance.
I doubt that's the reason - or at least not the main reason.
The main reason is legacy: Banks and other financial institutions are among the oldest users of computer systems (I would not hesitate to say that certain banks even utilized Hollerith tabulating systems; they certainly likely used tabulating and other accounting machines made by the early IBM).
Code in these institutions likely consists (in a large part) of many lines of COBOL (and possibly other archaic languages - I wouldn't doubt there is some old RPG lurking around).
You might ask "well, they could just translate the code or rewrite it" - but it probably isn't that simple. Over time, what we now call "technical debt" built up in a massive way. They fixed and patched and fixed some more, and probably comments don't match the code, and there likely isn't a singular design document or API description anywhere that shows what the system is actually doing (and any that does exist probably doesn't match what is actually happening). Coders came and went, most are probably dead now. But that code is the baseline for how the system works, warts and all. Heck - there's probably code in there that works around bugs in older COBOL compilers or who knows what else.
They keep this around - probably emulation on top of emulation - because to refactor all of this would take a super-human level of dedication and effort, and likely introduce new bugs along the way. In fact, there are likely still a ton of "edge case" bugs lurking deep inside that have yet to be found. It may be that fixing these bugs (if found during a refactor, say) might break other working code that depends upon them existing! Even if the refactor were successful, they would still only get to the point of having a system that does everything it currently does (but still may have unknown bugs).
You're right that there isn't a financial incentive to fix it. In fact, the opposite is likely, because if fixing it breaks something, in even a subtle way (and that would be worse than anything, because something like a small rounding error might go easily unnoticed until far too late) - it could end up costing the business so much money as to ruin them. I can't really blame them for being so hesitant about such a change.
This kind of legacy affects all kinds of businesses and institutions, especially the ones that are quasi-considered "foundational" to society, and hence are likely to be among the oldest users of computer, transaction processing, and accounting systems. A few others I can think of would be accounting firms, logistics (especially railroads - as they were among the first to use Hollerith's systems after the 1890 census), air traffic control (we've already seen this system try to be updated - and fail), airline ticketing (arguably maybe second oldest - how old is SABRE?), insurance...
None though - with the exception of the ATC systems - have the same level of repercussions to happen should a new system not perform and act the same as the old system(s) currently in place.
Big four employee outsourced to big bank. The lack of in-house knowledge is always troubling. Also screen scraping data and automating processes by having a program take control of the mouse/keyboard and click/type is a big seller to banks.
The effect of this strategy is similar to what happens in government - you have IT directors, who actually know very little about IT, but know they have $X to spend to solve problem Y. There is a number of companies who will gladly state they will solve problem Y for approximately $X. It creates this weird, self-reinforcing relationship.
My favorite lack of in-house knowledge is the mysterious but commonly used system acronym. You'd think "What does this stand for?" would be a question someone could always readily answer.
In Germany, we have something called SOFORT Überweisung which logs into your bank account to do online payments. As a non-German, I'm not sure how this is allowed, but it seems to be long-standing business. I don't think most of the banks like it, though.
It seems to me that many of the banks' concerns are legit.
I personally am not particularly keen on yet another API being made available to a bunch of rapacious VC-funded startups that facilitates my intimately personal financial info being spread around. Personally I bank with metro bank in the UK, they're new, competitive, and frankly pretty darn good, all for free. Do I need even more "financial services" thrown at me when there are apparently zillions of bricks and mortar operators already offering me everything from credit cards, subprime, insurance, car finance, re-mortgaging, yada yada yada yada ad infinitum?
I would argue that in this case, the banks' and my interests are somewhat aligned. They're protecting my privacy! How many profit seeking entities on the internet can say that?
I get it that fintech would dearly love to get its hands on people's money, but is there actual demand from said people for even more financial services? Or does this post simply amount to self-interested regulatory lobbying for a zero sum transfer of economic rents to the fintech space. Because you just know that if these APIs come about, fintech will find all sorts of dubiously "persuasive" ways to get people to allow access, possibly against their own interests.
You should be able to own your financial data, and access it in various ways.
Incumbent banks have been incredibly bad at creating modern digital user experiences. To analyse my spending, I have to login to my online bank with a fob and various passwords, download statements in .csv format (month by month, no batch download) and plot them in excel. It's not great.
I am really looking forward to more open banking API's. Monzo have one, a couple of others, but very few.
I have been working on a side project for the kind of banking experience I would want [1] - but it won't be possible without banks opening up API's.
Incumbent banks have been incredibly bad at creating modern digital user experiences. To analyse my spending, I have to login to my online bank with a fob and various passwords, download statements in .csv format (month by month, no batch download) and plot them in excel. It's not great.
That is certainly true, but it's not as if the shiny new fintech startups are necessarily any better. You'd think it would be business finance 101 to be able to reconcile money that a payment service collects from your customers and then deposits in your bank account with the money you expected them to be collecting, but unless there's been yet another API change lately, none of the services we use support a simple, reliable way to do this.
I bank with a major bank in the UK whose app is basically trash.
I have a credit card with a different provider
If I want a quick assessment of my accounts, either I need to log in to multiple accounts and add them together, or I can use an account aggregation service like yolt
Unfortunately nobody supports secure delegated auth, so you risk compromise.
There is very little difference between an API for you and an API for third parties. Unless the API just for you requires full credentials, and has no tiered access control, you could just give someone else the same API.
Eula's would be the greatest difference.
It is sad though, that no start-up would consider building a self-hosted version that'd keep all data local, instead wanting to offer it in SaaS form.
> It is sad though, that no start-up would consider building a self-hosted version that'd keep all data local, instead wanting to offer it in SaaS form.
i don't think that'd be automatically true. if there were enough security conscious users to create a market for a product like this, someone would provide.
The buisiness model really sucks though, because its hard to get people to pay for self-hosted stuff. On the other hand, people seem more amenable to a subscription.
This has the added benefit of possible adds and even gathering data of users.
The issue is exacerbated by the almost sure to be available open-source gratis software. Presuming the software is under-maintained.
Or more generally contracts and regulations, yes. If you need to be registered with some regulatory authority with security audit requirements, say, in order to be able to gain access to the API, then you cannot use the API, whether you would technically be able or not.
Its about efficiency. One Example: Being in the financial industry i have submit monthly reports with all my statements and activity. Luckily compliance software can automate this for me.
Currently, you'll need to be licensed by the Financial Conduct Authority in the UK to be able to use any of those APIs. That's a very high barrier to entry.
Teller does work but only a very small amount of users have production access at the moment. We've been very cautious about broadening access in case banks responded aggressively and generally speaking they haven't. A lot of work has been put into making the service reliable when it essentially sits on private APIs the banks prefer no one but them uses.
Teller will eventually support every bank but the banks we support currently are:
- Barclays
- Natwest
- Nationwide
- Santander
- RBS
- Ulster Bank
- Isle of Man Bank
Re the single blog post. We only blog when we have something to talk about that could potentially get to #1 on HN (as our only post did). We prefer to stay focused on building the product, but stay tuned for more posts soon. :)
Very interesting idea. Congrats on making it this far! I've been in banking for several years and there's lots of exciting new technology with no real use case. However, there's definitely many use cases that I can see for this one.
Thanks for the shout out! Open Bank Project is an open source API solution for banks. We have a catalogue of 130 standard pre-built banking APIs & 6000 developers worldwide using our interfaces. Also, PSD2 is the hook. There are many more APIs bank can open up which are outside the scope of PSD2.
In any case, Open Banking is definitely a global trend now, there are similar regulation efforts in the making in Australia, Singapore & South Korea.
Feel free to ask, if you have any question on this topic
security isn't the only problem. banks do not want the users to be fully informed. if you overdraw your account, they make more money.
personally, i'd love to write my own open source clients you can verify and compile yourself that provide better information and heuristics - but that'd hurt the banks.
there are several kinds of of users: those who make the bank money (the uninformed that do not care much) and those who are actually interested in their statements. the latter are loss leaders. for consumer banks, the latter seem to be in the majority.
an app that informs you with the best intention (warning: if you continue your spending for the rest of the month like you did this week, you'll pay a lot extra!) would shift the balance even more.
> Application Programming Interfaces (APIs) with a tokenized or alternative authentication method [...] can be inconsistent among financial institutions
That sounds like a made up argument. How is structured data less consistent than the scraping that is currently happening?
A more realistic argument is that banking systems are old and outdated, the cost of upgrade is enormous and no banks wants to be first.
Worse, some banks actually have standard APIs (e.g. OFX) than are used by commercial accounting softwares (e.g. mint, quicken) but the banks (e.g. Chase) charges a monthly fee to enable it.
I can tell banks plan or want to offer PSD2 services.
Either by a third party or offering them on there own.
What you might get though is a terible ugly interface to the inner part of the bank.
I also just read the newest version of the PSD2 proposal. There is (sadly or luckily) still no plan for further specification what a bank needs to offer or how.
But all in all PSD2 won't change the world of payment.it will be still easier to just use PayPal:)
I would like to see us abstract another layer between an arbitrary number and a person. We do have another system by which you enter a seemingly-random number to connect to the person, and we abstracted it with DNS. If you're a Facebook user, privacy concerns aside, you can change your number there and it gets updated on your friends' phones, I think. Is there any sort of OSS tool to do the same - automate updates to your contact info, so you don't have to tell your friends when you change the phone number which ingresses to your voice chat/SMS systems?
I don't see the benefit of this and it would introduce needless technical complexity; the UK's "faster switching" process is probably what you actually want for changing bank account.
Unless it has dramatically improved in the recent past, the UK's "faster switching" process isn't what any sane person would want for changing bank account. Last time I made the mistake of using it, it was a great way to swap spending a couple of hours notifying regular creditors and debtors of new account details with spending a couple of weeks cleaning up the mess the two banks involved made of the "automatic" transfer, and then spending the same time contacting all the same regular creditors and debtors to make sure they had the same new account details anyway. Has it got better lately?
How would that work? Your money doesn't just sit in your account. The banks use it to fund loans. Loans that they make money on. That's the premise of the whole finance system...
You are already allowed to leave your bank, and go to a different bank. The 'only' thing this would change is you would be able to take your account number with you.
BTW, european banks have explicitly not enabled this with IBAN, as the bank's name is now part of the account number.
In The Netherlands almost all banks support a "forwarding service" when switching banks: direct debits to the old account will be forwarded for at least a year to the new bank (account).
You can bet the banks didn't think of this themselves, but it was probably forced on them by a consumer protection law. It is, coincidentally, something most Dutch people don't know exists.
I don't think account numbers will get portable due to the issue you raised. However UK has kept pushing bank portability with "current account switch guarantee" which seamlessly transfers all your incoming/outgoing direct debits and the balance then closes your old account.
For people like me in the US, it's a gigantic pain to switch every direct-debit payment setting from the old routing/account number to the new.
Often with things like auto-bill-pay, you have to wait a billing cycle for the new changes to take effect. And you generally have to keep the two accounts open until everything is switched over, or face the wrath of failed direct debits.
I think that in europe this is less of a problem because european banking does not depend so much on direct-debits. But, on the other hand in Czech republic almost everybody uses service called SIPO (acronym translates to something like "Centralized direct-debit for citizen's payments") which works as additional virtual account for direct-debits which you can associate with whatever account you currently use (or even not associate to anything and get invoices in mail) without notifying anybody else.
Routing numbers aren't really something you can transfer, though. It defeats the whole point of a routing number if they get spread all over the system because clients move.
I created an app that would poll my transaction data from my bank's website and then send push messages to my phone if any new transaction comes in. I loved it. While doing groceries, having not left the line and packing up my goods, I'd get a push message of the transaction. This gave me a real feel for what happened on my bankaccount.
When notifo stopped (free API to send push messages without having to write your own native app); I didn't re-implement it, but from time to time, I wished I did; maybe something I should look into again.
Are you sure it's not already a feature your bank supports by now? All the major banks I know of (at least here in Canada) have supported this for awhile and can easily set up within a few clicks. I have SMS for transactions over $20.00 or whatever threshold I want and it's very nice getting a SMS seconds after paying or auto-withdrawals from my accounts.
It's not universal. My U.S. banks don't do this and only have email notifications, with a minimum threshold of $10 for some reason. My work account where I live abroad notifies me of every transaction and all attempts at logins on my online account. The pay terminal info is sent with the amount and current balance to my phone.
A lot of banks in the Middle East and Asia do this via SMS, as soon as the transaction is approved on the merchant's terminal you receive an SMS as confirmation.
This kind of thing would definitely guide my decision as to what bank I keep my money at. If my bank killed digit for example, I'd switch in a heartbeat.
I think big banks will learn pretty quickly that they are commodities, and they will need to compete on more than inertia alone.
And for this reason, there's the SimpleFIN spec and SimpleFIN Bridge[1]. Until banks implement the APIs themselves the SimpleFIN Bridge will bridge that gap.
Yoodle used to be the provider for Mint [1]. I assume they use some combination of authorized + non-authorized web-scrapping.
It looks like something that makes sense for big companies. The pricing [2] is a bit weird:
> Developers will find the Envestnet | Yodlee Aggregation API – our new and improved RESTful API architecture – easy to use, which will speed up integration and simplify data access. It will also enable developers to sign up on the developer portal for a $250 monthly fee and get immediate access to a testing and production environment.
So $250/mo just to sign up... no idea about what's the pricing per API call.
It's a crappy service, but the only one that support a wide range of banks in different countries:
- costs a lot: we pay around $20,000 / year
- we have a lot of support tickets from users who can't connect to their bank. We usually forward the issue to Yodlee and they fix it within a few days.
They scrap bank websites, but their algorithms do not seem very robusts. If the bank change a little detail in their UI, it breaks Yodlee's connector. Sometimes, the bank interface depends on user preferences (e.g. language, business/personal account), so the user has to configure his preferences before succeeding to sync Yodlee.
As an example; they have a PayPal interface too. Initially we thought of using them to connect to PayPal, but we noticed they scrapped the Paypal website too, instead of using their API. So, we had issues with some users who did not have a Paypal business account
Unfortunatelly, it's the only solution that supports a wide range of banks (~24,000 banks), so we are still using it. (for U.S. users, we use Plaid that is more reliable but support less banks)
Each bank has an open source adapter (this is different to Teller) and we encourage the community to help us building more adapters. So far we got adapters for 4 major banks in Spain, and 3 more are a work in progress.
Whether PSD2 is going to happen or not, we believe public APIs for banks will happen (even if banks don't like it).
IMHO banks arte not scared because of security concerns: they all have APIs in production already, they are just not documented. Their main concerns is basically how APIs used by third party services could affect their businesses.
[1] https://github.com/bankscrap/bankscrap