####################### Google Wave Attachments ####################### :Authors: Michael Lancaster :Version: 1.0 - May 2009 Wave messages may contain embedded binary attachments, such as images, PDF documents and ZIP archives. Because these binary files are qualitatively and quantitatively different to other wave content (rich text), they are handled as a somewhat special case within Google Wave. This document gives an overview on how attachments are represented within Google Wave, and how the servers interoperate to handle attachment uploading and serving. This whitepaper is part of a series. All of the whitepapers can be found on `Google Wave Federation Protocol site`_. .. _Google Wave Federation Protocol site: http://www.waveprotocol.org/whitepapers High level summary ################## Attachments are represented within a wave by an XML document, allowing changes (in upload progress, for instance) to be propagated to all users on the wave. Each attachment has a corresponding thumbnail image. For image attachments, this thumbnail is actually a small version of the image itself. In order to reduce latency for image attachments, HTML5 or Gears enabled clients may generate and upload a thumbnail before the image itself. For most other attachment types, the thumbnail is a generic representation of the attachment type (base on MIME type). Attachments are uploaded by the wave client using HTTP POST, and download of both attachments and their thumbnails is done using HTTP GET. Architecture ############ Attachment management is handled by a dedicated attachment server. This server is responsible for handling create, upload and download requests, generating thumbnails, reprocessing images, malware scanning, as well as for communications with the attachment store. The attachment server acts as an HTTP server (for handling attachment operations from the client), an RPC server (for handling attachment operations from internal agents, such as the mail gateway), and an RPC client for propagating attachment metadata to the wave server (see Google Wave Federation Architecture for details on the overall Google Wave architecture). .. image:: img/attachment-server-architecture.png Schema ###### Each attachment has a globally unique ID string, composed of the wave service provider domain, and a string that is unique for that provider. An example attachment ID for the wave sandbox wave provider would be "wavesandbox.com/3eb1c8ba-172b-4b1a-ae5b-d3140ed85c42". Each attachment is represented by a row in a replicated Bigtable (a Google proprietary scalable distributed database). The attachment metadata is represented by a protocol buffer stored in a column on that row. This protocol buffer contains such fields as the attachment size, upload progress, filename of the attachment, as well as a list of all wavelets that reference this attachment. The binary data for the thumbnail and attachment are each stored in separate columns on the same row. Thus an attachment row in the Bigtable looks like: AttachmentMetadata contains the metadata protocol buffer, ThumbnailData used to store the thumbnail BLOB (binary large object), AttachmentData used to store the attachment BLOB, for small attachments Large attachments are stored in a separate Bigtable for better storage efficiency. For performance, and simplicity of design, a subset of the attachment metadata is also copied to any wavelets which reference the attachment. Storing this metadata in the wavelet means that we don't have to do anything special at wave load time to ensure that the client has a copy of the attachment metadata. Whenever the attachment server makes a modification to the attachment metadata, it pushes out the change to all relevant wavelets (via RPC to the wave server(s)). This copy of the metadata is represented by an XML sub-document on a data document within the wavelet. The ID of the Data Document is based on the attachment ID such that there is exactly one attachment data document for each attachment on a given wavelet. The attachment metadata XML sub-document is defined by the following RNC (Relax NG) schema:: element attachment { attribute attachmentId { text }, attribute uploadProgress { xsd:integer }, attribute attachmentSize { xsd:integer }, attribute malware{ 0, 1 }, attribute stalled { 0, 1 }? // default = 0 attribute filename { text }, attribute mimeType { text }, attribute downloadToken { text }, element thumbnail { attribute width { xsd:integer }, attribute height { xsd:integer }, attribute clientGenerated { 0, 1 }? // default=0 }? element image { attribute width { xsd:integer }, attribute height { xsd:integer } }? } Changes to the attachment record are replicated to all waves which refer to that attachment. The blip in which the attachment was inserted also contains an XML node which references the attachment, located at the insertion point of the attachment. This XML element (known as the embed) is a placeholder for the thumbnail to be rendered and takes the form:: the thumbnail caption Attachment Creation ################### Attachments may be "created" in several different ways: * Uploading a thumbnail for the attachment * Uploading the attachment blob itself (or the first N bytes) * Linking an existing attachment to a new wave Each of these actions is represented by an attachment creation request. Attachment creation requests are sent as an HTTP POST, and may be either sent as an HTTP multipart request (enctype=multipart/form-data), or as a plain POST (enctype=application/x-www-form-urlencoded). The multipart POST is accepted to allow file uploads from non-Gears / HTML5 enabled browsers. In either case, the following fields may be sent either as HTTP POST parameters, or in the HTTP header:: required string attachmentId; required string waveletName; required int uploadType; // 0 for attachment, 1 for thumbnail optional bool complete; // true if data field represents the entire attachment optional int thumbnail_width; optional int thumbnail_height; For the non-multipart case, the filename is also optionally provided in the parameters / header.:: optional string fileName; and the bytes of the attachment / thumbnail are sent as the body of the POST. In the multipart case, only the part with name set to "uploadAttachment" is read, any other uploaded files are ignored. The filename is read from the filename field in the content-disposition for the file. Create requests are idempotent, so for instance it's okay to send one creation request with a thumbnail, and another with the first chunk of the attachment data. If the attachment record already exists, but the waveletName field does not correspond to any of the wavelets currently linked to the attachment, the existing attachment will be linked to the provided wavelet. Other fields which are already present in the existing attachment will be ignored. Example creation flow: 1. User initiates attachment creation by dragging an image into the browser (using Gears) 2. Client generates a globally unique ID for the attachment 3. Client thumbnails the image (using Gears) and displays it locally by adding an tag to the blip (other clients seeing the tag will display an empty thumbnail frame). The client then sends an HTTP POST containing a create request, and the thumbnail data, to the Attachment server (via WFE) 4. Attachment server creates a record in permanent storage for the attachment and stores the (re-encoded for security) user-provided thumbnail 5. Attachment server returns success to the client 6. Attachment server creates a data document on the wavelet and adds a copy of the attachment metadata. 7. Thumbnail is now ready to download 8. Client sends an HTTP POST containing the attachment 9. Attachment server updates the attachment record in permanent storage 10. Attachment server returns success to the client 11. Attachment server generates a thumbnail for the attachment 12. Image attachments are reprocessed to prevent XSS attacks, and attachments are scanned for malware 13. Attachment server updates the attachment data document on the wavelet 14. Attachment is now ready to download Steps 8-14 may happen in parallel with 3-7. Below is an example of a multipart (non-Gears) creation request:: POST /wfe/upload/result HTTP/1.1 Host: wave.google.com Content-Type: multipart/form-data; boundary=---------------------------10102754414578508781458777923 Content-Length: 195197 -----------------------------10102754414578508781458777923 Content-Disposition: form-data; name="uploadAttachment"; filename="Downtown.pdf" Content-Type: application/pdf -----------------------------10102754414578508781458777923 Content-Disposition: form-data; name="waveletName" wavesandbox.com/w+6bf32acc-bd29-45c2-a252-699af690f5a6/conv+root -----------------------------10102754414578508781458777923 Content-Disposition: form-data; name="attachmentId" wavesandbox.com/3eb1c8ba-172b-4b1a-ae5b-d3140ed85c42 -----------------------------10102754414578508781458777923 Content-Disposition: form-data; name="uploadType" 0 -----------------------------10102754414578508781458777923-- Uploading ######### Clients may upload large attachments in multiple chunks using an upload request:: required string attachmentId; required int offset; optional int fullSize; The binary data is sent as per the creation request. Either multipart or form POSTs are accepted. An upload request may not be sent until the upload request (or create request) for the previous chunk has been acknowledged. That is, we don't currently support pipelining. Chunks must not overlap. Behaviour is not specified if chunk boundaries overlap. The response to HTTP upload / create requests is a string containing a single JSON object of the form:: { responseCode: , errorMessage: " " } Possible values for the responseCode field are:: 0 (OK) 1 (INVALID_TOKEN) 2 (INVALID_REQUEST) 1000 (INTERNAL_SERVER_ERROR) The errorMessage field will not be provided for the non-error case (OK). Otherwise, it will contain a human-readable (although not necessarily end-user friendly) error message. In conjunction with these custom error codes, HTTP response codes should also be respected, however, due to limitations with cross-domain POSTs, the JSON response codes are used in preference. Attachment / Thumbnail download ############################### A download request takes the following form:: required string attachmentId; required string downloadToken; Requests for thumbnails / attachments are sent on different URLs, but otherwise look identical. The response to these requests is an HTTP response containing the bytes of the attachment / thumbnail, with the HTTP Content-Disposition header set to "attachment". The mime type of the response is set to the mime type of the attachment or thumbnail. Authentication / Authorization ############################## Google web-apps use a centralized cookie-based authentication system. Authentication for upload and creation requests uses this system. In order to write the corresponding attachment data document into an associated wavelet, the user must be a participant on that wavelet. Downloads are authenticated using a download token which is stored in the attachment data document on the wavelet. Thus to download an attachment or a thumbnail, the user must at some point in time have had access to both the attachment id and the download token. Duplicate elimination ##################### Because we expect a large percentage of attachments to be duplicates, we have an offline de-duping procedure. We store a weak hash with each attachment, and an offline process indexes attachments by hash, detects collisions, and then does a byte-by-byte comparison to eliminate duplicates. This is only done on attachments that are completely uploaded, and effectively immutable, and only on 'large' blobs, which are stored in a separate store. We maintain a level of indirection for these large blobs, so that we don't have to update the pointers upon duplicate detection and to prevent the leakage of information about the existence of previously uploaded attachments. References ########## * E. Nebel and L. Masinter, `Form-based File Upload in HTML `_, IETF RFC 1867, November 1995 * F. Chang et al., `Google Research Publication: Bigtable `_, OSDI'06: Seventh Symposium on Operating System Design and Implementation, November 2006. * S. Lassen and S. Thorogood, `Google Wave Federation Architecture `_, June 2009