- Where: zoom.us
- When: February 11, 17:00-18:00 UTC
- Location: link on calendar invite
- Contact:
- Name: Lin Clark
- Email: [email protected]
None required if you've attended before. Email Lin Clark to sign up if it's your first time.
The meeting is open to CG members only. You can join the CG here.
The meeting will be on a zoom.us video conference. Installation is required, see the calendar invite.
- Opening, welcome and roll call
- Please help add your name to the meeting notes.
- Please help take notes.
- Thanks!
- Announcements
- Sumbit a PR to add your announcement here
- Proposals and discussions
- Discussion: What are the next steps in adapting WASI to use Interface Types handles and resources?
- Sumbit a PR to add your agenda item here
Pat Hickey Andrew Brown Sergey Rubanov Lin Clark Mark McCaskey Till Schneidereit Dan Gohman Josh Dolitsky Mingqiu Sun Ralph Squillace Nicholas Radu Matei Matt Fisher Johnnie Birch Luke Wagner
Lin Clark: Are there any announcements? (none)
Lin Clark: Lets talk about how we’re going to move from this version of WASI to a new version that uses handles
Dan Gohman: Witx has been designed to anticipate interface types. It has String and Handle types. These are pretty close to the ones in interface types - PR 391 takes things to the next level and evolves the rest of the witx types towards the interface types. Going forward we need to get rid of the witx pointer and get buffers to work the way they will in interface types, but we’re still waiting to hear from interface types the final word on how those will work. See Interface Types issue 68. Our job on the witx side is to evolve into the Interface Types buffer types. Without waiting we can add support for strings and return types and lower them into the existing C ABI that WASI programs use.
Dan Gohman: Another piece is WASI modularization. It has been in progress for a while but several pieces are coming together. Issue 378. Describes a more granular include mechanism than witx uses right now - import a symbol from a module, rather than import a file.
Dan Gohman: The next two big pieces, which we can address in detail in the future: How do you pass handles into programs? Currently there is a table with preopens in it, but interface types is going to give us new mechanisms that are better than the preopen concept.
Dan Gohman: The last piece is defining new abstractions. wasi PR 240 describes how to merge the send and recv on sockets with read and write on files. We can generalize part of files and sockets into a concept of I/O streams. In interface types you’ll have the vocabulary to ask for e.g. a stream you can write to, without knowing or caring that its backed by a socket or file or pipe etc. What is the noun that you use in function signatures to describe an I/O Stream? Beyond I/O Streams there’s non-streaming operations on files which don’t work on sockets or pipes - pread/pwrite - there’s a place for an I/O object with non-stream personality as well, an array of bytes or a we can call it a dynamic array because it can be resized. There will be other concepts as well - key-value stores, windowed IO with buffers... At the moment the nascent WASI I/O proposal will evolve to develop these ideas. There will still be files, and then a function that gives you a stream from a file. Files are going to be required to interop with existing systems, but we can push people towards interfaces that don’t tie them to the complications of those existing systems, particularly file semantics are very difficult to compose but streams are. This will help us enable shared-nothing linking of modules - we don’t want modules to have to share a filesystem to talk to each other. Questions?
Ralph Squillace: What kinds of objections to this direction do you anticipate?
Dan Gohman: This is going off the path of POSIX. Existing applications and programming languages will be more work to port to I/O streams than having us port libc and standard libraries to WASI. We have ideas for making the standard io operations work on top of I/O streams - you can treat an i/o stream as a libc FILE if you only ever read and write from it, other operations give an error.
Ralph Squillace: The philosophical discussion is about how you weigh the objectives of different applications moving to WASI - some will have to adapt specific POSIX-isms to WASI, where does the work of porting existing code break down - does the code stay in WASM and not use WASI, have its own custom runtime - or does it move to WASI in the coherent way that resolves the semantic differences. I don’t personally see myself trying to port an existing huge monolithic application to WASI, whereas I do see myself working on libraries to make them work in WASI.
Lin Clark: How much is WASI filesystem still going to allow porting POSIX applications?
Dan Gohman: It will allow it. Its a question of priorities rather than absolutes. WASI today runs plenty of interesting POSIX code that doesnt do things like fork(2), WASI filesystem is going to still be there for these applications to use. But we don’t want that to be part of the default for WASI, we want to encourage applications to use simpler concepts that don’t imply a whole filesystem. I want to build tooling that bridges the gap between the shared-nothing-linking space and making big applications work.
Till Schneidereit: We can look at this in terms of layering - there is a core of WASI that may not apply to every use case but you can virtualize many of the other needs on top of it. You can add a filesystem ion top of the core, in many cases, and let applications that need it use it.
Dan Gohman: In terms of migration from where WASI is today to this ideal system - the WASI I/O abstractions can be worked on now, we have witx where we need it. We can flesh out exactly how streams and dynamic arrays work today.
Dan Gohman: Another aspect is portability. I’ve been working on support for WASI on Windows vs Unix the last few months - there are so many differences between those platforms, the semantics of filesystems is radically different and there’s no portability in many cases (its too expensive to bridge them). The WASI I/O abstractions can be designed to hide all of the non-portable parts of filesystems. We can get away from having to specify “what does windows do here” and instead specify the things an application actually needs.
Ralph Squillace: Layering will require thought.
Pat Hickey: The customers we have writing applications right now aren’t concerned with the differences between the filesystem guarantees on linux vs windows, they care about http requests and responses. If we have I/O abstractions that unify the way http bodies work with the way certain aspects of the filesystem work, that is better for the applications we care about.
Till Schneidereit: There are lots of use cases that work with streaming, which has traditionally been a file or a socket, but with abstractions that are hard to generalize. My interpretation of what you were just saying is that if we had a way to have a cleaner system that didn’t have that incidental platform dependence, it would be much easier to build code that works across these very different environments. We have all these applications that are popping up that are very different. Ideally we have a layer that doesn’t care about host differences, and then the layer that cares about that. What we do is just one specific example of that. Don’t want to optimize for old use cases, but all of the new novel ones that are coming about.
Radu Matei: Is the assumption that wasi-filesystem will be developed in parallel?
Dan Gohman: Yes, WASI-filesystem will continue, and wasi-io will be a peer.
Ralph Squillace: Drawing older threads in, what about BSD sockets PR.
Dan Gohman: The contributor who created that hasn’t been active, but I expect that will fit into the system in the same way as wasi-fs. You could get a stream. Networking is a complex topic and I dont’ want to simplify, but the first step will be to take sockets and say the socket gives you a stream that you can read, a stream you can write, and bi-directional r/w.
Async is a subtlety that we’re leaving out. We can say that async is an orthogonal concern. We expect the Wasm platform will give us better answers over time. But the fundamental concepts of I/O are independent of whether you block that I/O.
Lin Clark: Any other thoughts/comments
Luke Wagner: I have a hypothesis I want to test. Why are people going to want to use wasm outside of a browser? Some people think it’s great because of ISA. We don’t think that’s a great motivator. We think running guest code in latency sensitive and low trust scenarios is the motivator. Things like routers ____ would be host APIs. The libraries would be wasm. These problematic cases don’t have a high value reason to be ported to Wasm.
Ralph Squillace: I understand where you’re coming from. I don’t see WASI as a monolithic process sandbox. I do think that the idea about the sweet spot is right. I’m not a big believer in the thought that the world’s code is going to all be running on WASI. But I think it’s worth consideration how much of the world’s knowledge can be run as WASI. Shouldn’t wall off for all time, but I think the way that Dan laid out the direction, it lets us start with the sweet spot and either layer the others, or add support to WASI later on. I would hate to frame the boundary ruling out knowledge transfer.
Luke Wagner: I agree with your points, so the layering and virtualization are also motivated by that.
Dan Gohman: I can add one thing. Thinking of lib-c, and lots of languages have lib-c like std lib: under the covers, something that people are using as a file could be represented by streams. We can make that work under the covers. We could talk about how that will exactly work. But this is the framework. We can make porting easier by allowing you to use streams by allowing you to use files, but discourage from using file stat and depending on i node.
Andrew Brown: It feels to me that filesystem should be in phase 3 or 4. How do these changes affect moving those to later stages
Dan Gohman: That’s a great question. Wasi-fs is waiting to figure out it’s place. This conversation moves that forward. Understanding that wasi-fs won’t attempt to be abstracting over all hosts, wasi-io does that, and accepting that’s ok means that wasi-fs can probably move to phase 3 at this point.
Till Schneidereit: I think the better way to put it is that it’s not an either/or, we don’t need to make it completely host specific, but let’s find a balance that’s actually viable.
Radu Matei: do we have a document?
Dan Gohman: Will be writing that up. Wanted to introduce here first. Will write up wasi-io ideas soon
Pat Hickey: One blocker to wasi-fs advancing. The filesystem has cloud ABI system of rights. Need to work out what to do with that.