script.module.hay A distributed cloud cache mechanism based on dispersy

Kodi community has a serious problem like zombie users, consuming Terrabytes of data from free services like, tmdb, tvbd, anidb, muscibrainz etc free metadata providers for their scrapers and 3rd party addons.

Most of those services already came to a decision of shutting down their services because of this serious problem, and luckily they still go on but keep deleteing keys of some their api users. However they still need donations to go on. So please do so Smile

There is also another aspect of that problem that most of the endusers are doing this without having any idea, and there is a bill to pay at the end of the day.

I have been developing this addon in mid-term to solve this specific problem.

script.module.hay is a key-value storage system that you throw a need in to your db (hay) and find it after you need it again, so you dont need to query again from remote (lets say tmdb), a simple cacher. The needle can be a pythonic object, hay module will serialize it and store it anyways, can be json or pickle serializer does not matter. This part is already implemented and there is not much interesting stuff about it.

Here comes the twist, when there is no cached entry in the local database, “hay” module will ask the distributed community for that specific needle, whenever someone has it, “hay” will download the data from the peer instead of service provider. The magic here is there is no need for portforwarding, no need for p2p backend like torrent, dispersy is just message based protocol and has nothing to do with torrent and has been proved to scale about thousands or millions without issues. And the shared files will be max 100kb or something json or pickled ascii / binary data. If it can not find anything in the distributed community, it will download from the service, and next time all the neighbours in the community wont need to.

With this implementation, service providers like tmdb will save bandwith i am guessing about %80 at least, kodi users will receive good and continious service with little bandwith contribution, (without even knowing). The creators of the dispersy at delft university will have the wildest environment on the internet (kodi community Smile) for scientific purposes. win-win-win.

This all is my interpretation but delft university guys are quite warm about kodi addons, but for successfull implementation all scrapers must move on to dispersy comunity with “hay” addon, so i would really appreciate what would the community’s reaction would for such a solution?

further reading:
https://dispersy.readthedocs.io/en/devel/