- koolba 9 months ago
If I'm understanding how this works, the idea is to compile a program locally for x64, extract out the .text, convert it to hex, and have a Python UDF shim in Redshift execute it.
Pretty nifty! The surprising thing for me isn't getting it to play chess though. It's that Amazon let's you run arbitrary binaries from within Redshift. I would have figured that the Python UDFs would be locked down to prevent this type of thing.
: User defined function
- MichaelBurge 9 months ago
It's not Redshift-specific. A similar function should work in Postgres, though it might be slightly more dangerous since it wouldn't be wrapped in a container like Redshift's is.
But this technique isn't useful for Postgres, because you'd just write an ordinary C extension at that point. That's why I focus on Redshift in the article.
> As of PostgreSQL 7.4, PL/Python is only available as an "untrusted" language, meaning it does not offer any way of restricting what users can do in it. It has therefore been renamed to plpythonu. The trusted variant plpython might become available again in future, if a new secure execution mechanism is developed in Python. The writer of a function in untrusted PL/Python must take care that the function cannot be used to do anything unwanted, since it will be able to do anything that could be done by a user logged in as the database administrator. Only superusers can create functions in untrusted languages such as plpythonu.
- slig 9 months ago
Thank you for writing this post. Learned a lot.
- xyzzy_plugh 9 months ago
If it's like other databases, the Python UDF probably runs as a subprocess with limited abilities.
> Amazon Redshift blocks all network access and write access to the file system through UDFs.
- kylegalbraith 9 months ago
Wow this is somewhat surprising that you got this to work. I didn't really know Redshift was capable of this. It will take me some time to fully digest your post, but it is quite an interesting find none the less.
- johansch 9 months ago
Quite neat.. but I kind of long back to the time where computing hacks didn't use relatively expensive pay-per-usage cloud services run by Jeff Bezos.
- matchagaucho 9 months ago
Depends on how you value your time.
The nickels I pay to spin up Redshift are nothing compared to the time it'd take for me to provision, install, configure, and host all the various components.
- johansch 9 months ago
Obviously it is not about the actual few dollars, but the principle.