SML

SML (Structured Email)

SML (Structured Email) is an emerging standard for adding machine-readable data to email messages. It enables developers to create more intelligent email applications that can understand and act on the content of messages, rather than just treating them as plain text. By implementing SML, developers can create more sophisticated email experiences, such as interactive elements, automated actions, and smarter filtering based on the actual content of messages.

Brainstorm Use cases and UI representation

Examples of possible use cases for SML:

1. Poll
2. Finding a meeting time that suits most people, or the most important participants. UI: Could show my calendar appointements at the proposed times. Optionally deny all proposals where I have a conflicting appointment.
3. Location tracking UI: show on map. Possibly with trace (past locations) and direction (where heading). Open in Google maps, allowing me to route there.
4. email verification Button does http call, no browser page then deletes message
5. invoice, order confirmation sent to accounting application puts products and price in a database of pitched products
6. delivery notification when linked to order and/or invoice by ID, can show overall status of each ordered product , including order, delivery status, dates, price, etc currently scattered in at least 4 emails

What is currently done via email, but manually? How can we automate it? What is currently not done by email, but whatsapp etc? What could be done, what nobody does right now?

• Order Management: When receiving an meeting invitation, seeing whether you already have appointments can show overall status of each ordered product, including order, delivery status, expected delivery time, dates, price, vendor information, custom notes, etc. Currently, this information is scattered in at least 4 emails, or even more when orders are split or merged during delivery.

• Repairs: Guarantee. Terms. Call service technician.
• Related products: Toner for printer, headphones/case for phone

• Parcels
  • When receiving tracking number, use package sender API to get latest package status

Purchase Order

• Can accept contract via email. Email client knows who you are.
• Resellers have an interest in automating this. Recipents, too. Automatic offers. Offer acceptance by 1 click. Auto process acceptance -> Order system.
• Company: 2 HTTP REST calls
  • To create PO, reply to Request: Product + Quantity -> Price (including rebate)
  • To make order: Product + Quantity + Price, Customer data -> Order
• Coupon
• Manage subscriptions

• Show list of subscriptions that I pay monthly/yearly.
• When it renews, when to cancel etc., make event to cancel at a certain time.
• Compare PayPal subscriptions.
• Show all emails related to that subscription

• Tell my contacts that I moved to a new home/business address.

• Ditto for email address or chat
• Update companies that I have a contract with. Authentication using the data (customer/contract numbers) in the original customer welcome email. (Include in the human-readable text part a hint that this contains structured email, to convince companies to accept this.)

• Email encryption

• Request email pubkey.
• Company welcome email requests customer email pubkey.
• Company can send invoices encrypted
• Company can accept customer email signed and as authenticated
• Increases security, for customer and company.

• Signup: Prefill fields
• Location sharing between friends: Facilitate real-time location sharing for efficient meetups.
• Real time updates in a privacy-sensitive way, using server chosen by sender. Allow sender to choose how long to send updates: Not, 1 hour, 1 day, forever.
• UI: Show person on map. Possibly with trace (locations in the last few minutes) and direction (where heading).
• Open location in preferred routing app (Google Maps etc.), allowing me to route there.
• Also: Location destination (complement to street address), navigation (from/to)
• Calendar invitation (exists, standard, but complex): Propose meeting event. Invitee can respond with Yes/No/Maybe
• Poll: Create a poll system for anything that needs opinion of others, similar to what

WhatsApp and other apps do.

• Event Organization: Agree on the best time for meetings of a larger group. Integrates with the calendar: Can show which options conflict with existing meetings in my calendar, and easily reject those. When an option is chosen, add it to my calendar. Similar to Calendarly etc., but without external web site needed. Directly integrated in the mail. No need to sync my calendar to a third party service, therefore more privacy sensitive.
• Processing email bounces, vacation notifications and other automatic mail.
• Email login verification: Button does http call, no browser page. Then automatically deletes the message.

When in the browser form (see below), user agent can send email, without waiting for email from site. web form can contain token that the user sends to the site. User is in the form and confirms that he wants it to be sent. user can stay in the form without going to email app.

• Travel Coordination

• After booking a flight, adding the event to the calendar.
• When the date is approaching, showing local transport options to/from airport, e.g. using Google Maps.
• Show weather at target location
• Compare KDE Itinerary app.
• Flight: Change time zone in calendar app, for that day range only.

• Event tickets: Like event + location + song
• Links

• Show small preview of the web page
• Fetch page, find Structured Data in the page (e.g. recipe, spotify song etc.), extract the data.
• UI
• Show it inline in the email (possibly with expander)
• Show it on the sidebar
• with an applet or app (see Song)
• Store as bookmark, with title, URL, and structured data

• Song

• play the song with your preferred song app. Even if you send me a spotify link, I can still play it in Apple Music.
• Follow the artist. Show new songs of that artist.

• News / RSS
• COVID vaccination yes/no with confirmation/signature
• Issue and ticketing system

• Reply to comment
• Close issue, assign, change priority

• Refresh information
• Forms

• Compare AMP, but doesn't have meaning, Form is a JSON.
• Request Data object (schema.org), contains fields
• UI infers form from the data types
• Update form before load, with server sender via HTTP.
• Already submitted
• Show what other people already submitted, e.g. polls.
• Add meaning (SML) to each field. This allows the client to auto-fill the fields.
• Mobile
• Doesn't support web extensions
• webviewer in mail client
• replace mobile browser
• Use cases
• Polls
• Rating
• Comment
• Ticket system changes
• Appointment, e.g. doctor's appointment, with extra data

• Discovery of form via HTTP, after entering email address

1. Patient enters doctor as recipient
2. Email client does .well-known request to web server, which offers a number of forms. Each recipient may have a different list of forms.
3. Patient selects the form
4. Patient enters form fields and sends
5. Doctor system can auto-process the request, e.g. appointment

• Requesting appointment with doctor, person, restaurant
• On web forms, display link to email with form, advertizing that I can send this by email

• email client can prefill form values
• calendar can show availability

• Resource booking

• Meeting room
• Book it
• Call maintenance staff
• Get info about equipment
• Has QR code -> webpage

• When receiving links
• whitelist of sites that offer schema.org meta data in their web pages
• UI: Offer expander after link. When clicked, fetch webpage, extract schema.org data and offer corresponding UI.
• When sending links (shared or pasted)

• fetch webpage, extract schema.org data and add it as SML

https://www.audriga.eu/test/sml-web/#/external?visual_debug=true&single=true&source=https://www.tagesschau.de/ausland/europa/ukraine-kinder-besatzung-100.html

https://miro.com/app/board/uXjVLWh1v_c=/

Alternatives

• PKPass (Apple) - Wallet, entry tickets, coupons
• "Fancy web forms in email" (both support refresh of email content)
  • AMP Email (Google) (not the same as AMP for websites)
  • Actionable Messages (Microsoft) https://amdesigner.azurewebsites.net/

Deployment

• When sending structured email, include in the human-readable text part a hint that this contains structured email and to change email clients.
• Process emails using JS - converts HTML into structured email. Addons. MIT license.

URI schemes

• Fallback
• Browser or OS have fallbacks hardcoded, for each URI scheme
• Fallback URL in URI schemes, generic
• Fallback in HTML: <a href="geo:57474" hreffallback="https://maps.google.com/57474">

https://searchfox.org/mozilla-central/source/toolkit/components/extensions/schemas/extension_protocol_handlers.json

Webbrowser

<structured data="JSON with schema.org">Fallback UI</structured> tag

Webextension (in browser, installed by mail client) renders <structured> as iframe.

SML Forms (fallback: <form>)

• Can immediately prefill data. No privacy/phishing, because iframe, so site has no access. Smoother user interaction. Only need to click submit.

Music

• Show user's preferred music player inline in the page, to play the song. Play via Spotify, Apple Music, local player etc. Player UI in the page.

PWA: WebShare

• register webapp in Android for "share" button
• register mimetypes

Specs

• HTTP endpoints:
  • GET, POST with auth token
  • mimetype: "application/ld+json"
  • Later: Also allowed: "image/png". Multi-document upload, e.g. form + camera picture?
• Forms: Site only requests structured data object from schema.org. Mapping done by User Agent, mapping data type to input fields.
  • Can be used in email or browser
  • Form attributes
    • requested data type from schema.org
    • Submission URL, HTTP POST, see above
    • Authentication token, see above
• Refresh
  • Refresh URL, HTTP GET, see above
  • Refresh frequency
  • Expiry
  • Use cases: Polls, Calendar, Location sharing
• <structured> HTML tag
• Discovery
  • https://<emaildomain>/.well-known/structured/person/<emailaddress>/
  • Returns: List of forms -> URL, which returns form content

How to find more use cases

• What regular tasks are currently routinely done via email, but manually? How can we

automate it?

• What is currently done by WhatsApp etc, but not email?
• What could be done, what nobody does right now?
• In what other aspects of users routines there is still space for automation? What other ideas do we have? Feel free to add any other possible use case!