Available Lua Libraries¶

When Sandbox is disabled all standard Lua modules are available; with a Sandbox ON (default) only some of them can be used. See Standard Library for more.

Splash ships several non-standard modules by default:

json - encoded/decode JSON data
base64 - encode/decode Base64 data
treat - fine-tune the way Splash works with your Lua varaibles and returns the result.

Unlike standard modules, custom modules should to be imported before use, for example:

base64 = require("base64")
function main(splash)
    return base64.encode('hello')
end

It is possible to add more Lua libraries to Splash using Custom Lua Modules feature.

Standard Library¶

The following standard Lua 5.2 libraries are available to Splash scripts when Sandbox is enabled (default):

Aforementioned libraries are pre-imported; there is no need to require them.

Note

Not all functions from these libraries are currently exposed when Sandbox is enabled.

json¶

A library to encode data to JSON and decode it from JSON to Lua data structure. It provides 2 functions: json.encode and json.decode.

json.encode¶

Encode data to JSON.

Signature: result = json.encode(obj)

Parameters:

obj - an object to encode.

Returns: a string with JSON representation of obj.

JSON format doesn’t support binary data; json.encode handles Binary Objects by automatically encoding them to Base64 before putting to JSON.

json.decode¶

Decode JSON string to a Lua object.

Signature: decoded = json.decode(s)

Parameters:

s - a string with JSON.

Returns: decoded Lua object.

Example:

json = require("json")

function main(splash)
    local resp = splash:http_get("http:/myapi.example.com/resource.json")
    local decoded = json.decode(resp.content.text)
    return {myfield=decoded.myfield}
end

Note that unlike json.encode function, json.decode doesn’t have any special features to support binary data. It means that if you want to get a binary object encoded by json.encode back, you need to decode data from base64 yourselves. This can be done in a Lua script using base64 module.

base64¶

A library to encode/decode strings to/from Base64. It provides 2 functions: base64.encode and base64.decode. These functions are handy if you need to pass some binary data in a JSON request or response.

base64.encode¶

Encode a string or a binary object to Base64.

Signature: encoded = base64.encode(s)

Parameters:

s - a string or a binary object to encode.

Returns: a string with Base64 representation of s.

base64.decode¶

Decode a string from base64.

Signature: data = base64.decode(s)

Parameters:

s - a string to decode.

Returns: a Lua string with decoded data.

Note that base64.decode may return a non-UTF-8 Lua string, so the result may be unsafe to pass back to Splash (as a part of main function result or as an argument to splash methods). It is fine if you know the original data was ASCII or UTF8, but if you work with unknown data, “real” binary data or just non-UTF-8 content then call treat.as_binary on the result of base64.decode.

Example - return 1x1px black gif:

treat = require("treat")
base64 = require("base64")

function main(splash)
    local gif_b64 = "AQABAIAAAAAAAAAAACH5BAAAAAAALAAAAAABAAEAAAICTAEAOw=="
    local gif_bytes = base64.decode(gif_b64)
    return treat.as_binary(gif_bytes, "image/gif")
end

treat¶

treat.as_binary¶

Get a binary object for a string.

Signature: bytes = treat.as_binary(s, content_type="application/octet-stream")

Parameters:

s - a string.
content-type - Content-Type of s.

Returns: a binary object.

treat.as_binary returns a binary object for a string. This binary object no longer can be processed from Lua, but it can be returned as a main() result as-is.

treat.as_string¶

Get a Lua string with a raw data from a binary object.

Signature: s, content_type = treat.as_string(bytes)

Parameters:

bytes - a binary object.

Returns: (s, content_type) pair: a Lua string with raw data and its Content-Type.

treat.as_string “unwraps” a binary object and returns a plain Lua string which can be processed from Lua. If the resulting string is not encoded to UTF-8 then it is still possible to process it in Lua, but it is not safe to return it as a main result or pass to Splash functions. Use treat.as_binary to convert processed string to a binary object if you need to pass it back to Splash.

treat.as_array¶

Mark a Lua table as an array (for JSON encoding and Lua -> JS conversions).

Signature: tbl = treat.as_array(tbl)

Parameters:

tbl - a Lua table.

Returns: the same table.

JSON can represent arrays and objects, but in Lua there is no distinction between them; both key-value mappings and arrays are stored in Lua tables.

By default, Lua tables are converted to JSON objects when returning a result from Splash main function and when using json.encode or ref:splash-jsfunc:

function main(splash)
    -- client gets {"foo": "bar"} JSON object
    return {foo="bar"}
end

It can lead to unexpected results with array-like Lua tables:

function main(splash)
    -- client gets {"1": "foo", "2": "bar"} JSON object
    return {"foo", "bar"}
end

treat.as_array allows to mark tables as JSON arrays:

treat = require("treat")

function main(splash)
    local tbl = {"foo", "bar"}
    treat.as_array(tbl)

    -- client gets ["foo", "bar"] JSON object
    return tbl
end

This function modifies its argument inplace, but as a shortcut it returns the same table; it allows to simplify the code:

treat = require("treat")
function main(splash)
    -- client gets ["foo", "bar"] JSON object
    return treat.as_array({"foo", "bar"})
end

Note

There is no autodetection of table type because {} Lua table is ambiguous: it can be either a JSON array or as a JSON object. With table type autodetection it is easy to get a wrong output: even if some data is always an array, it can be suddenly exported as an object when an array is empty. To avoid surprises Splash requires an explicit treat.as_array call.

Adding Your Own Modules¶

Splash provides a way to use custom Lua modules (stored on server) from scripts passed via HTTP API. This allows to

reuse code without sending it over network again and again;
use third-party Lua modules;
implement features which need unsafe code and expose them safely in a sandbox.

Note

To learn about Lua modules check e.g. http://lua-users.org/wiki/ModulesTutorial. Please prefer “the new way” of writing modules because it plays better with a sandbox. A good Lua modules style guide can be found here: http://hisham.hm/2014/01/02/how-to-write-lua-modules-in-a-post-module-world/

Setting Up¶

To use custom Lua modules, do the following steps:

setup the path for Lua modules and add your modules there;
tell Splash which modules are enabled in a sandbox;
use Lua require function from a script to load a module.

To setup the path for Lua modules start Splash with --lua-package-path option. --lua-package-path value should be a semicolon-separated list of places where Lua looks for modules. Each entry should have a ? in it that’s replaced with the module name.

Example:

$ python3 -m splash.server --lua-package-path "/etc/splash/lua_modules/?.lua;/home/myuser/splash-modules/?.lua"

Note

If you use Splash installed using Docker see Folders Sharing for more info on how to setup paths.

Note

For the curious: --lua-package-path value is added to Lua package.path.

When you use a Lua sandbox (default) Lua require function is restricted when used in scripts: it only allows to load modules from a whitelist. This whitelist is empty by default, i.e. by default you can require nothing. To make your modules available for scripts start Splash with --lua-sandbox-allowed-modules option. It should contain a semicolon-separated list of Lua module names allowed in a sandbox:

$ python3 -m splash.server --lua-sandbox-allowed-modules "foo;bar" --lua-package-path "/etc/splash/lua_modules/?.lua"

After that it becomes possible to load these modules from Lua scripts using require:

local foo = require("foo")
function main(splash)
    return {result=foo.myfunc()}
end

Writing Modules¶

A basic module could look like the following:

-- mymodule.lua
local mymodule = {}

function mymodule.hello(name)
    return "Hello, " .. name
end

return mymodule

Usage in a script:

local mymodule = require("mymodule")

function main(splash)
    return mymodule.hello("world!")
end

Many real-world modules will likely want to use splash object. There are several ways to write such modules. The simplest way is to use functions that accept splash as an argument:

-- utils.lua
local utils = {}

-- wait until `condition` function returns true
function utils.wait_for(splash, condition)
    while not condition() do
        splash:wait(0.05)
    end
end

return utils

Usage:

local utils = require("utils")

function main(splash)
    splash:go(splash.args.url)

    -- wait until <h1> element is loaded
    utils.wait_for(splash, function()
       return splash:evaljs("document.querySelector('h1') != null")
    end)

    return splash:html()
end

Another way to write such module is to add a method to splash object. This can be done by adding a method to its Splash class - the approach is called “open classes” in Ruby or “monkey-patching” in Python.

-- wait_for.lua

-- Sandbox is not enforced in custom modules, so we can import
-- internal Splash class and change it - add a method.
local Splash = require("splash")

function Splash:wait_for(condition)
    while not condition() do
        self:wait(0.05)
    end
end

-- no need to return anything

Usage:

require("wait_for")

function main(splash)
    splash:go(splash.args.url)

    -- wait until <h1> element is loaded
    splash:wait_for(function()
       return splash:evaljs("document.querySelector('h1') != null")
    end)

    return splash:html()
end

Which style to prefer is up to the developer. Functions are more explicit and composable, monkey patching enables a more compact code. Either way, require is explicit.

As seen in a previous example, sandbox restrictions for standard Lua modules and functions are not applied in custom Lua modules, i.e. you can use all the Lua powers. This makes it possible to import third-party Lua modules and implement advanced features, but requires developer to be careful. For example, let’s use os module:

-- evil.lua
local os = require("os")
local evil = {}

function evil.sleep()
    -- Don't do this! It blocks the event loop and has a startup cost.
    -- splash:wait is there for a reason.
    os.execute("sleep 2")
end

function evil.touch(filename)
    -- another bad idea
    os.execute("touch " .. filename)
end

-- todo: rm -rf /

return evil