How To Run a Server with Pyodide¶

Pyodide provides a version of the CPython interpreter compiled for WebAssembly, allowing you to execute Python programs either in a web browser or in NodeJS.

This guide outlines how to run your pygls server in such an environment.

Important

This environment imposes some restrictions and limitations to consider. The most obvious restrictions are:

only the STDIO method of communication is supported
threads are unavailable, so your server cannot use the @server.thread() decorator
while it is possible to use async-await syntax in Pyodide, pygls does not currently enable it by default.

The setup is slightly different depending on if you are running your server via the Browser or NodeJs

Using NodeJS¶

The most likely use case for using NodeJS is testing that your server works in Pyodide without requiring the use of a browser testing tool like Selenium. In fact, this is how we test that pygls works correctly when running under Pyodide.

To help illustrate the steps required, we will use pygls’ test suite as an example.

Tip

You can find the complete setup in the tests/pyodide folder of the pygls repository.

Writing our Python code as normal, each server is executed with the help of a wrapper script:

$ node run_server.js /path/to/server.py

The simplest wrapper script might look something like the following

const fs = require('fs');
const { loadPyodide } = require('pyodide');

async function runServer(serverCode) {
    // Initialize pyodide.
    const pyodide = await loadPyodide()

    // Install dependencies
    await pyodide.loadPackage("micropip")
    const micropip = pyodide.pyimport("micropip")
    await micropip.install("pygls")

    // Run the server
    await pyodide.runPythonAsync(serverCode)
}

if (process.argv.length < 3) {
    console.error("Missing server.py file")
    process.exit(1)
}

// Read the contents of the given `server.py` file.
const serverCode = fs.readFileSync(process.argv[2], 'utf8')

runServer(serverCode).then(() => {
    process.exit(0)
}).catch(err => {
    process.exit(1);
})

The above code is assuming that the given Python script ends with a call to your server’s start_io() method.

Redirecting Output¶

Unfortunately, if you tried the above script you will find that your language client wouldn’t be able to establish a connection with the server. This is due to fact Pyodide will print some log messages to stdout interfering with the client’s communication with the server:

Loading micropip, packaging
Loaded micropip, packaging
Loading attrs, six
Loaded attrs, six
...

To work around this in run_server.js we create a function that will write to a log file.

const consoleLog = console.log
const logFile = fs.createWriteStream("pyodide.log")

function writeToFile(...args) {
    logFile.write(args[0] + `\n`);
}

And we use it to temporarily override console.log during startup

async function runServer(serverCode) {
    // Annoyingly, while we can redirect stderr/stdout to a file during this setup stage
    // it doesn't prevent `micropip.install` from indirectly writing to console.log.
    //
    // Internally, `micropip.install` calls `pyodide.loadPackage` and doesn't expose loadPackage's
    // options for redirecting output i.e. messageCallback.
    //
    // So instead, we override console.log globally.
    console.log = writeToFile
    const pyodide = await loadPyodide({
        // stdin:
        stderr: writeToFile,
    })

    await pyodide.loadPackage("micropip")
    const micropip = pyodide.pyimport("micropip")
    await micropip.install("pygls")

    // Restore the original console.log
    console.log = consoleLog
    await pyodide.runPythonAsync(serverCode)
}

While we’re redirecting output, we may as well also pass the writeToFile function to pyodide’s stderr channel. That way we’re also able to see the server’s logging output while it’s running!

Important

Since node’s fs API is asynchronous, don’t forget to only start the server once the log file has been opened!

logFile.once('open', (fd) => {
    runServer(serverCode).then(() => {
        logFile.end();
        process.exit(0)
    }).catch(err => {
        logFile.write(`Error in server process\n${err}`)
        logFile.end();
        process.exit(1);
    })
})

Workspace Access¶

Using the Browser¶

WebWorkers¶

Running your language server in the browser’s main thread is not a great idea since any time your server is processing some message it will block the UI. Instead we can run the server in a WebWorker, which we can think of as the browser’s version of a background thread.

Using the monaco-editor-wrapper project, connecting your server to the client can be as simple as a few lines of configuration

import '@codingame/monaco-vscode-python-default-extension';
import { MonacoEditorLanguageClientWrapper, UserConfig } from 'monaco-editor-wrapper'

export async function run(containerId: string) {
  const wrapper = new MonacoEditorLanguageClientWrapper()
  const userConfig: UserConfig = {
    wrapperConfig: {
      editorAppConfig: {
        $type: 'extended',
        codeResources: {
          main: {
            text: '1 + 1 =',
            uri: '/workspace/sums.txt',
            enforceLanguageId: 'plaintext'
          }
        }
      }
    },
    languageClientConfig: {
      languageId: 'plaintext',
      options: {
        $type: 'WorkerDirect',
        worker: new Worker('/run_server.js')
      },
    }
  }

  const container = document.getElementById(containerId)
  await wrapper.initAndStart(userConfig, container)
}

Where run_server.js is a slightly different version of the wrapper script we used for the NodeJS section above.

Overview¶

Initializing Pyodide¶

The initPyodide function is fairly similar to the runServer function from the NodeJS example above. The main differences are

We are now redirecting stderr to console.log rather than a file
We are now also redirecting stdout, parsing the JSON objects being written out and passing them to the postMessage function to send them onto the client.
We are not calling server.start_io in our server init code.

async function initPyodide() {
    console.log("Initializing pyodide.")

    /* @ts-ignore */
    let pyodide = await loadPyodide({
      stderr: console.log
    })

    console.log("Installing dependencies.")
    await pyodide.loadPackage(["micropip"])
    await pyodide.runPythonAsync(`
        import micropip
        await micropip.install('pygls')
    `)

    // See https://pyodide.org/en/stable/usage/api/js-api.html#pyodide.setStdout
    pyodide.setStdout({ batched: (msg) => postMessage(JSON.parse(msg)) })

    console.log("Loading server.")
    await pyodide.runPythonAsync(`<<insert-your-server-init-code-here>>`)
    return pyodide
}

Initializing the Server¶

Since we are not calling the server’s start_io method, we need to configure the server to tell it where to write its messages. Ideally, this would be done by calling the set_writer() method on the server’s protocol object.

However, at the time of writing there is a bug in Pyodide where output is not flushed correctly, even if you call a method like sys.stdout.flush()

To work around this, we will instead override one of the protocol object’s methods to output the server’s messages as a sequence of newline separated JSON strings.

# Hack to workaround https://github.com/pyodide/pyodide/issues/4139
def send_data(data):
    body = json.dumps(data, default=server.protocol._serialize_message)
    sys.stdout.write(f"{body}\n")
    sys.stdout.flush()

server.protocol._send_data = send_data

The above code snippet should be included along with your server’s init code.

Handling Messages¶

Finally, with the server prepped to send messages, the only thing left to do is to implement the onmessage handler.

const pyodidePromise = initPyodide()

onmessage = async (event) => {
    let pyodide = await pyodidePromise
    console.log(event.data)

    /* @ts-ignore */
    self.client_message = JSON.stringify(event.data)

    // Run Python synchronously to ensure that messages are processed in the correct order.
    pyodide.runPython(`
        from js import client_message
        message = json.loads(client_message, object_hook=server.protocol.structure_message)
        server.protocol.handle_message(message)
    `)
}

The above handler

Converts incoming JSON objects to a string and stores them in the client_message attribute on the WebWorker itself
Our server code is then able to access the client_message via the js module provided by Pyodide
The server parses and handles the given message.