Omar Shehata's writing

Creating a DOOM-inspired aesthetic with PlayCanvas

Sat, 31 Jul 2021 00:00:00

Last week I made a game called Death's Clutch in 10 days for a game jam hosted by Newgrounds. Making it was a lot of fun because it involved a lot of customized rendering like making DOOM-inspired 2D billboards that move in 3D, or hacking PlayCanvas's material system to get stylized lighting.

I hope this "how it's made" article will be useful for others seeking to stylize their PlayCanvas games, and I'm curious if any readers know of better ways to implement these effects (especially curious about comparing with other engines/workflows).

Why does the same note sound different across instruments?

Sun, 28 Feb 2021 00:00:00

I really wanted to answer this question for myself from first principles. This ended up being a fun exploration of timbre and Fourier transforms.

Unraveling the JPEG

Wed, 01 May 2019 00:13:00 GMT

Learn about image compression by looking at the raw image bytes and how they get encoded/decoded.

Linear Discriminant Analysis

Mon, 03 Dec 2018 00:13:00 GMT

LDA is a technique for working with higher dimensional data that was always presented analytically to me, and I wanted to see it visually.

The Mystery of the Corrupted MIDI File (Untangling an encoding mess with JavaScript Typed Arrays)

Tue May 22 2018 00:13:00 GMT

Learning how to debug a corrupted MIDI file takes us down a journey into binary data, encoding, and typed arrays.

A Beginner's Guide to Coding Graphics Shaders

Wed Apr 15 2015 00:13:00 GMT

Shader programming can come off as enigmatic black magic, and is often misunderstood. There are many code samples out there that demonstrate incredible...

Dall e telephone

Fri, 06 Sep 2024 00:00:00

Office snapshot

Mon, 10 Jun 2024 00:00:00

Rosanos gift

Thu, 16 May 2024 00:00:00

My new radio

Wed, 21 Feb 2024 00:00:00

Scenes from egypt

Wed, 01 Nov 2023 00:00:00

Outdoor cats & epistemology

Fri, 15 Sep 2023 00:00:00

Glimpse into recent news in egypt

Mon, 07 Aug 2023 00:00:00

Debugging double tap in godot

Sun, 07 May 2023 00:00:00

Mapping my ithaca

Sat, 03 Dec 2022 00:00:00

My newsletter

Thu, 10 Nov 2022 00:00:00

Biking in alaska

Sun, 31 Jul 2022 00:00:00

Tell yourself better stories

Mon, 25 Jul 2022 00:00:00

My number forms

Sat, 23 Jul 2022 00:00:00

Improv math song

Sat, 09 Jul 2022 00:00:00

My favorite memory palace episodes

Wed, 08 Jun 2022 00:00:00

Seattle sunset

Fri, 15 Apr 2022 00:00:00

Notes on infinite powers

Mon, 04 Apr 2022 00:00:00

Project ideas

Tue, 01 Feb 2022 00:00:00

Notes on 10 percent happier

Tue, 28 Dec 2021 00:00:00

Let it be genre defining

Mon, 04 Oct 2021 00:00:00

A still more glorious dawn awaits

Thu, 16 Sep 2021 00:00:00

Recreating light art

Mon, 13 Sep 2021 00:00:00

Do it even if you dont feel like it

Mon, 13 Sep 2021 00:00:00

Cync - exploring non quantized music

Sat, 07 Aug 2021 00:00:00

Behind the scenes of deaths clutch

Sat, 31 Jul 2021 00:00:00

Do what is hard

Thu, 08 Jul 2021 00:00:00

The hidden gallery

Sat, 05 Jun 2021 00:00:00

How to turn an image black and white

Mon, 12 Apr 2021 00:00:00

The best zoom call

Sun, 04 Apr 2021 00:00:00

Playing celeste is like learning the piano

Mon, 22 Mar 2021 00:00:00

Why I dislike movie previews

Sun, 21 Mar 2021 00:00:00

Studying the night of the exam

Tue, 02 Feb 2021 00:00:00

An ode to snow

Sun, 20 Dec 2020 00:00:00

Visualizing the solar analemma

Mon, 07 Dec 2020 00:00:00

On inspiration and determination

Tue, 01 Dec 2020 00:00:00

A preface to learning english

Wed, 25 Nov 2020 00:00:00

WebGL / WebGPU Jan 2023 meetup highlights

Omar Shehata — Wed, 01 Feb 2023 17:18:26 GMT

The Khronos WebGL/WebGPU meetup is a recurring online event where the folks working on these web graphics API’s share updates followed by talks from the community.

Here I’ve summarized my personal highlights & takeaways from this week’s meetup.

Recording link: https://www.youtube.com/watch?v=Jl06sOvMnvU

For my past event recaps, see April 2022, January 2022, July 2021.

WebGL Updates

Ken Russell, Khronos Group
Slides

Some of the things Ken was most excited to share:

Wide color gamut support coming to WebGL — allows you to make use of newer displays with more vibrant color spaces
Proposed Pixel Local Storage extension — it potentially has “dramatic performance improvements for many use cases”. Can follow its development here.
Provoking vertex extension — can improve performance a lot for applications using flat shading.
Proposed polygon offset clamp extension — easy to use extension that reduces many artifacts in real time shadows

Ken also shared that they are very close to shipping WebGL’s ANGLE/Metal backend. This is exciting because it’ll allow WebGL applications to continue being fully supported on the latest Metal API across MacOS & iOS.

I found this to be a good article on what pixel local storage is. It allows you to write per-pixel data to a fast on-chip storage, removing the need to write to a buffer for many lighting techniques. So that uses less memory and is also much faster.

The OpenGL wiki has a good explanation for what the “provoking vertex” is, see also a discussion on it in WebGL.

WebGPU Updates

Kelsey Gilbert, Mozilla
Slides (same as above)

v1.0 is almost there! Currently targeting 2023 Q1. Expecting release around Chromium version 113 (see release schedule) on Windows, ChromeOS, and MacOS (and Linux and Android later)
Lots of progress on using WebGPU outside the browser: (1) Deno has built in WebGPU support (2) Dawn, a cross platform implementation of WebGPU, is “99% on par with Chromium” and has Node bindings

Kelsey called for community support for helping develop WebGPU. The best ways to contribute right now are (1) providing feedback on the API & general usage through GitHub or Matrix chat, she emphasized this is now the time when feedback would be most impactful & helpful (2) help writing conformance tests.

ThreeJS & WebGPU

Ricardo Cabello, Google

Mrdoob talked about how ThreeJS has been working towards supporting WebGPU. The biggest obstacle to making it a drop-in replacement for WebGL is that WebGL shaders must be written in GLSL and WebGPU shaders are WGSL.

The reason this is a problem for ThreeJS is that many developers use material.onBeforeCompile to edit the generated shader GLSL code before it is compiled, to create custom materials. WebGPU would be a breaking change for this because you’d need to rewrite your modifications in WGSL.

“NodeMaterial to the rescue!” says Mrdoob. Instead of writing your materials in GLSL or WGSL, or any shading language, you write it in this higher level node system, and it’ll be easy for the engine to generate the right shader for the renderer/platform.

Below is an example code snippet (source) for this iridescence material demo.

Source code link

https://threejs.org/examples/?q=gltf#webgl_loader_gltf_iridescence

They’ve also built a nodes material editor playground you can use to interactively create and edit these kinds of materials:

https://threejs.org/examples/?q=nodes#webgl_nodes_playground

Mrdoob mentioned that they’ve been looking at MaterialX (https://materialx.org/) as an open standard for representing these kinds of materials. They’ve been trying to generally align how they design the materials in ThreeJS with this standard so they aren’t reinventing everything from scratch.

MaterialX already has a lot of support in various engines & authoring tools, and ThreeJS now has a loader for it. This is exciting because it means you can export your models from Blender & Maya etc and get them to look exactly the same on the web with ThreeJS.

A final resource Mrdoob shared was NodeToy, an interactive playground where you can create, edit, and share materials with a node editor. It even generates ThreeJS code or GLSL code for any of the materials on the site.

https://nodetoy.co/

Mrdoob mentioned there’ll still be an option to write your own shaders if you want to, but for most use cases NodeMaterial will be the way to go.

A technical journey through Google Earth

John Anderson, Google

John talked about the evolution of Google Earth’s technical architecture over the years, starting with this awesome sneak peak of the very first time they tried running it with Emscripten and AsmJS:

Originally Google Earth only worked in Chrome, running native code with PNaCL. This is the reason they always had a “launch” button earth.google.com instead of being able to launch directly to the app.

The switch to a fully WebAssembly version happened in 2020, and allowed it to work in all browsers. In 2021 the “launch” button was removed. Google Earth now runs on a shared C++ renderer across Web, Android, and iOS.

There were two really interesting optimizations that John explained:

Google Earth uses JavaScript <canvas> for correct text rendering

Instead of trying to include a whole font/text engine in the WASM application, which would have been too big and too difficult to maintain, they instead rely on the web browser which can already do this. So to render text, the app draws the text into a JS canvas, reads it back, and then renders that inside of the 3D Google Earth scene.

2. Runtime performance was significantly improved by doing a directly copy from <canvas> to a WebGL texture

What they were initially doing for the text rendering was:

Draw into a canvas
Read the canvas as bytes, send it to the WASM app
WASM app creates a WebGL texture with these bytes, and uploads it to the GPU

To speed this up, they:

Draw into a canvas
Copy the canvas image data directly to a WebGL texture, give the WASM app a reference to that texture

That reduced a lot of FPS jank.

An example of this kind of direct copy in WebGPU, see: https://gpuweb.github.io/gpuweb/#dictdef-gpuimagecopyexternalimage

Q&A

There was a lot of interesting discussion at the Q&A at the end. One question in particular that stood out to me was on whether we can expect a “devtools” debugger/inspector for WebGPU. It sounded like the answer was yes, there’s a lot of interest both from the WebGPU spec writers and the browser vendors for providing good debugging tools.

Kelsey mentioned debug groups & labels which is one tool that WebGPU has today that allows you to label parts of your pipeline to make it easier to debug when things go wrong. You can read more about it here: https://gpuweb.github.io/gpuweb/explainer/#errors-errorscopes-labels

Ken closed out the session answering a question about how WebGPU is going to raise the bar of performance of realtime graphics on the web much closer to native applications.

“The future is bright,” he said. And I agree!

I hope you found this useful! You can find me on Twitter or on my website.

You can sign up to be notified of the future WebGL/WebGPU meetups here: https://www.khronos.org/news/subscribe/

Better outline rendering using surface IDs with WebGL

Omar Shehata — Tue, 11 Oct 2022 09:23:13 GMT

This article builds on my previous “How to render outlines in WebGL” tutorial. I’ll explain an improved technique that eliminates most artifacts of the previous article, and gives you more control over where exactly the lines are drawn in your geometry.

Below is an example of the artifacts with the previous technique (left) on the windows of the ship that are fixed using the newer method (right).

I learned about this from Ian MacLarty’s twitter thread about how outline rendering works in Mars First Logistics:

Screenshot from Mars First Logistics.

Live demo & source code

Try out this outline rendering implementation in the demo below. You can drag and drop your own models into the viewer (as a single .glb/glTF file) or log in with Sketchfab and paste a model URL.

https://threejs-outlines-postprocess.glitch.me/

Source code for this demo: https://github.com/OmarShehata/webgl-outlines/tree/main/threejs

See also a minimal version without any of the debug parameters: https://github.com/OmarShehata/webgl-outlines/tree/main/threejs-outlines-minimal

How it works — an overview

The previous technique uses a combination of the normals & depth of the scene to detect edges in a post process pass. The new technique uses “surface IDs” instead of the normals.

Pipeline image from the previous “How to render outlines in WebGL” article

We compute surface IDs for each mesh at startup time (or offline). We assign a globally unique ID for each “surface” of a mesh (more on that below). These ID’s are stored as a vertex attribute, rendered to a buffer, and used as an input for the edge detection post process.

Here is a simple example where the normals are not sufficient to detect an edge:

The front face of the box has the same normal as the wall behind it, so we can’t detect edges in certain camera angles where they align.

Now let’s take a look at the version where each surface has a unique ID (here I’m visualizing these ID’s by assigning a unique color to each one):

We can always detect an edge here.

Looking back at the ship example, we can see that this is why the outlines disappear on the windows of the ship as it rotates. Below is the normal buffer visualized on the left, and the surface IDs buffer visualized on the right:

The normals of the window interior align with the body of the ship for a few frames, so we are no longer able to detect an edge, while the depth difference isn’t sufficient to detect the edge either.

Whereas with the surface IDs, the two surfaces always have distinct values assigned to them, regardless of the camera angles.

How to compute surface IDs of a mesh

Ian MacLarty defines a surface as “a set of vertices that share triangles”. That means what we’re doing here is looking for all connected components in our mesh and assigning each a unique ID.

Let’s see how this works on a simple example. Here I have a cube in Blender with the top face selected, and the index of each vertex shown.

The top face of the cube is made of 2 triangles, and 6 vertices. The vertices “1” and “2” are shared between these two triangles, so we consider them part of the same “surface”.

If we take a look at another face of the cube, we’ll see that they do NOT share any vertices with the top face:

This means vertices “4” and “1” both sit right on top of each other, and so do “5” and “0”.

Why is the cube constructed this way? We technically only need to define 8 vertices total for this cube if we removed all duplicate vertices that are on top of each other like this. Instead, this cube has a total of 24 vertices (each face has 4 distinct vertices, and we have 6 faces).

The reason geometry is often constructed this way is because it allows us to define normals that are stored at each vertex. Vertices 0 and 1 have normals that point up, and vertices 4 and 5 have normals that point sideways.

Below are the normals visualized with the cube having 24 vertices (left) and if we combined them into 8 vertices (right). We can’t store all the normals correctly in the version on the right.

Going back to our goal of finding distinct surfaces on the mesh: we’re using the fact that any triangles that are physically connected & face the same direction can all share vertices. So if we treat our mesh as a graph, with edges as links that connect vertices, then we can traverse this graph looking for connected components. This is what the FindSurfaces.js class does.

One important caveat here is the word “can” in the sentence: “any triangles that are physically connected & face the same direction CAN all share vertices”. It’s entirely possible that your mesh has triangles that should be sharing vertices but are not. In that case a false edge will be rendered. We’ll look at this case in the “Tweaking your geometry” section below.

Using the surface IDs in the edge detection post process

Once we have the surface ID’s assigned to each vertex, we store them as a vertex attribute. We then have a render pass to render all surface ID’s in the scene to a buffer.

When drawing surface ID’s to the screen, I found the best results are achieved by:

Normalizing them between 0 and 1
Writing it to a float texture

I found a texture type of HalfFloat to be sufficient precision. The shader for writing the surfaceID to a buffer can be found in FindSurfaces.js.

In the edge detect post process, once we detect a non-zero difference between the surface ID value of one pixel to its neighbors, we consider that an edge. This happens in CustomOutinePass.js.

The FindSurfaces class keeps track of a “max surface ID” that is set as a uniform when rendering the surface ID buffer so that normalizing the surfaceID happens in the shader. This allows new geometry to be added to the scene dynamically without having to go back and update the vertex attributes of all existing geometry.

Tweaking your geometry for outline rendering

Because this method of detecting edges relies on how the geometry is constructed, you may find artifacts caused by how your geometry is constructed.

To me this is one of the biggest advantages of this technique: you have very fine control over exactly which edges in your model should be part of the outline rendering when authoring it. You don’t have to fiddle with your edge detection parameters that may fix one case in one model but produce artifacts in another model. It is extra effort to fix these by hand but it’s a good tradeoff for getting high quality outline results.

Let’s take a look at an example using the Tugboat model (originally from Google Poly). Originally the windows looked like this:

We’re seeing the 2 triangles that make up the windows, instead of the whole thing as one square. We can verify that this is a geometry problem by looking at the “Surface ID debug buffer” mode which shows us a unique color for each surface ID:

We want both triangles in the windows to have the same surface ID.

All we need to do is merge the vertices for these 2 triangles. I’ve had some trouble figuring out the best way to do this in Blender (even if I merge the vertices, the exported glTF often still has separate vertices for the triangles). My current method is to merge the triangles themselves:

Select the two (or more) faces in edit mode that should be considered one surface

2. Switch to vertex mode to automatically selected all the vertices attached to those faces

3. Press “M” to merge vertices, select “By Distance”. The distance can be very small since we only want to merge vertices that are right on top of each other

4. Finally, switch back to face selection mode, right click on the selected face, and click “Dissolve faces”. This combines the 2 triangles in each window into one face, and ensures each window gets 1 surface ID computed.

Notes on other ways to compute surface IDs

Instead of manually fixing your geometry by merging vertices, you could pre-compute and bake in a value in your model authoring workflow.

Christopher Sims talks about his method of getting perfect outlines by using a Blender add on called IDMapper. This lets you bake in unique IDs across your mesh as vertex colors, and also makes it easy to edit them. I believe this is partially automatically generated.

What I’d love to see is a completely automatic method that lets you mark surfaces based on a threshold, in the same way the ThreeJS EdgesGeometry works. That class looks over all triangles, and checks the angle they make with the triangles they’re connected to. You could use that information to merge vertices for triangles that are all relatively flat. And you’d be able to tweak this threshold per model.

Edit: I’ve put together a tool to automate this, but turns out automation some drawbacks, namely texture coordinates may get distorted: https://github.com/OmarShehata/webgl-outlines/tree/main/vertex-welder#vertex-welder

In any case I think it’s clear that the best outline results come from methods that do some form of geometry processing instead of relying just on screen space information, since that can be much more consistent across camera angles while also giving you more control over all edge cases (pun intended).

Thanks for reading! If you found this helpful, sign up to my newsletter to follow my work & stay in touch.

If you have any suggestions or corrections to the code or technique, open an issue on GitHub (https://github.com/OmarShehata/webgl-outlines/) or reach out to me directly. You can find my contact info at: https://omarshehata.me/

How to use WebGPU Timestamp Query

Omar Shehata — Mon, 29 Aug 2022 09:44:10 GMT

This tutorial covers how to use WebGPU’s timestamp query to measure how much time your GPU commands take to execute.

Timestamp queries are an optional WebGPU feature and may not be supported in all implementations. At the time of writing it is currently disabled in browsers for security reasons (see discussion here).

Summary

Below is a summary of the workflow for using timestamp queries:

Request access to timestamp-query when initializing the device
Create a query set with capacity N (number of timestamps you want to store in a frame)
Create a storage buffer with size N * 8. This is where the timestamp results are stored in nanoseconds, 64 bit numbers.
Record timestamps by calling commandEncoder.writeTimestamp. This will record a timestamp after all previous commands have finished.
Call commandEncoder.resolveQuerySet to write the recorded timestamps to the storage buffer
Copy the results from the storage buffer to the CPU and decode them as a BigInt64Array (see BigInt)

Example: you want to measure how long it takes to render all transparent objects in your scene. You would write 1 timestamp before any transparent draw calls, then a 2nd timestamp after all the transparent draw calls, then compute the difference.

Step by step guide

A full example implementation can be found in this PR: https://github.com/OmarShehata/webgpu-compute-rasterizer/pull/5/files

0 — Enable timestamp queries

Launch Chrome with the following command line flag:

--disable-dawn-features=disallow_unsafe_apis

1 — Queryset & buffer setup

First we add timestamp-query to the list of required features when requesting the device:

const device = await adapter.requestDevice({
	requiredFeatures: ["timestamp-query"],
});

This will throw with the following error if it’s not enabled/supported:

Uncaught (in promise) TypeError: Failed to execute 'requestDevice' on 'GPUAdapter': Unsupported feature: timestamp-query

Then we create a query set and a query buffer:

const capacity = 3;//Max number of timestamps we can store
const querySet = device.createQuerySet({
	type: "timestamp",
	count: capacity,
});
const queryBuffer = device.createBuffer({
	size: 8 * capacity,
	usage: GPUBufferUsage.QUERY_RESOLVE 
	  | GPUBufferUsage.STORAGE
	  | GPUBufferUsage.COPY_SRC
	  | GPUBufferUsage.COPY_DST,
});

2 — Write timestamps

We call commandEncoder.writeTimestamp(querySet, index) at any point in the pipeline where we want to record a timestamp:

// Add timestamps in between GPU commands
commandEncoder.writeTimestamp(querySet, 0);// Initial timestamp
// commandEncoder.draw(...)
commandEncoder.writeTimestamp(querySet, 1);

The index must be less than or equal to capacity - 1 that we defined earlier.

3 — Resolve timestamps to buffer

At the end of your frame, call commandEncoder.resolveQuerySet to actually write the timestamps to the storage buffer:

commandEncoder.resolveQuerySet(
	querySet, 
	0,// index of first query to resolve 
	capacity,//number of queries to resolve
	queryBuffer, 
	0);// destination offset

4 — Read the results

To get the timestamp results we need to copy the queryBuffer data to the CPU. Reading from a WebGPU buffer is explained in more detail here: https://web.dev/gpu-compute/#commands-submission.

Once you have the buffer data, you decode it as a BigInt typed array. The timestamps are recorded in nanoseconds.

// === After `commandEncoder.finish()` is called ===
// Read the storage buffer data
const arrayBuffer = await readBuffer(device, queryBuffer);
// Decode it into an array of timestamps in nanoseconds
const timingsNanoseconds = new BigInt64Array(arrayBuffer);

// ....

async function readBuffer(device, buffer) {
  const size = buffer.size;
  const gpuReadBuffer = device.createBuffer({size, usage: GPUBufferUsage.COPY_DST | GPUBufferUsage.MAP_READ });

  const copyEncoder = device.createCommandEncoder();
  copyEncoder.copyBufferToBuffer(buffer, 0, gpuReadBuffer, 0, size);

  const copyCommands = copyEncoder.finish();
  device.queue.submit([copyCommands]);

  await gpuReadBuffer.mapAsync(GPUMapMode.READ);
  return gpuReadBuffer.getMappedRange();
}

(Optional) 5 — Create labels

To make the output more useful, we can define labels for each timestamp we collect, and then print out the diff so we can see the time it took for each step in our pipeline. It may also be useful to convert the result to milliseconds.

This is implemented in the reference example here: https://github.com/OmarShehata/webgpu-compute-rasterizer/pull/5/files.

Hope you found this helpful! Sign up to my newsletter to follow WebGL/WebGPU writing & stay in touch.

Big thanks to Markus Schütz whose Potree implementation WebGPU implementation provided a great example of how to use timestamp queries. Thanks to Yang Gu for explaining how to enable it in Chrome.

WebGL / WebGPU April 2022 meetup highlights

Omar Shehata — Sun, 01 May 2022 19:58:16 GMT

The Khronos WebGL/WebGPU meetup is a recurring event where the folks working on these web graphics API’s share updates followed by talks from the community.

I’ve summarized highlights from April’s event in this article.

For my past event recaps, see January 2022, July 2021.

WebGL and WebGPU Updates

Ken Russell — Google
Kelsey Gilbert — Mozilla
Slides
Timestamp, 1:20

Wide color gamut support for WebGL is being implemented in Chrome & Firefox (example: P3 support, see proposal here). I believe this is planned for WebGPU in the future, see discussion here.
Lots of recent Firefox WebGL updates, most notably support for OffscreenCanvas (enables real multithreaded WebGL, can render on workers)
Chrome & Safari working to switch to Metal backend. You can test Chrome with Metal today (set “Choose ANGLE graphics backend” to “Metal” in about:flags). Report bugs in the ANGLE issue tracker.
Current WebGPU 1.0 spec release is targeted for Q3 2022. Breaking changes are still expected. Main things they’re working on now are updates to the shading language (reducing verbosity, adding features requested by community etc), privacy & security, and conformance tests.
Ways to contribute to WebGPU: (1) use the API and report bugs. (2) Write conformance tests (great way to learn WebGPU) (3) contribute samples/demo

You can get in touch with the folks working on the WebGPU spec through (1) the GitHub repo (2) the Matrix chat room. Ken & Kelsey emphasized that questions, feedback, and contributions are welcome!

One of the new WebGPU resources Kelsey shared was Tim Jones’ shader playground: https://shader-playground.timjones.io/. This lets you see the output of various shader compilers, or see how GLSL -> WGSL transpiling works. It can be a useful tool for learning or debugging.

Below is an example of a GLSL snippet transpiled to WGSL using Naga.

Intel’s Updates on Web Graphics

Yang Gu — Intel Web Graphics Team
Slides
Timestamp, 18:45

Yang talked about Intel’s contributions to web graphics, dating back to Nov 2011. Their recent focus has been on WebGPU, and TensorFlow.js’s WebGPU backend.

The Intel Web Graphics team are currently the maintainers of the TensorFlow.js’s WebGPU backend. He mentioned that they are already seeing better performance than the WebGL backend with medium & large size models.

Yang also mentioned that there is a large community of web graphics developers in China and that they are trying to grow this community. If you’re interested in that, Yang Gu’s contact is in the slide below.

3D Tiles Next: Bringing massive 3D geospatial data to the web

Sean Lilley — Cesium
Slides
Timestamp, 28:25

Sean talked about 3D Tiles, a format used for visualizing & storing 3D geospatial data. He explained that it’s a “spatial data structure for glTF, where each tile is a glTF. It allows you to stream in only what’s needed for a given view”.
3D Tiles Next is an upcoming revision of this format that has (1) better support for metadata (2) and a cleaner integration with the glTF ecosystem, which makes it easier to use 3D Tiles in engines that support glTF.
The Cesium team has been working on 2 new glTF extensions to support this: (1) EXT_mesh_features (2) EXT_structural_metadata. Together these extensions allow you to embed rich metadata into glTF models while still allowing you to combine & batch geometry for fast rendering.
Sean covered a little bit about how the CesiumJS renderer works. How they handle global scale precision, how they batch draw calls for fast rendering, and how they handle picking features in city scale datasets.

Below is an example of the metadata support Sean mentioned. The glTF nodes in this 3D Tileset have a “feature ID texture”, that allows the engine to look up properties associated with a particular point on the mesh. This is often used to encode information from an AI model (like that it classified it as a window, or a door etc) or information from the sensors that capture & generated this model (such as accuracy, camera angle, etc).

Creative RAM Savings in WebGL

Toni Martí — Tribia
Slides
Timestamp, 50:21

Toni shared his “data textures” technique for aggressively reducing GPU-RAM consumption in WebGL (by about 80% in his application)
He implemented this in xeokit — a viewer for high precision 3D engineering & BIM models on the web
The basic idea is using NO vertex attributes. Instead, all vertices, colors etc are written to textures. This uses significantly less memory because color data doesn’t need to be repeated for all vertices that use it. This also makes it faster to update.
He covered the implementation of this technique in this PR. Specifically, see the vertex shader in TrianglesDataTextureColorRenderer.js

Below is the demo he showed: 73k objects with individual color & position offsets that are updated 10 times per second.

I hope you found this useful! You can find me on Twitter or on my website.

You can sign up to be notified of the future WebGL/WebGPU meetups here: https://www.khronos.org/news/subscribe/

WebGPU January 2022 meetup — takeaways

Omar Shehata — Tue, 01 Feb 2022 21:12:01 GMT

WebGPU January 2022 meetup — takeaways

The WebGL/WebGPU January talk is up (https://www.youtube.com/watch?v=wYAvVUFQP2M)! I encourage you to watch full video.

I’ve compiled my takeaways here (with a focus on WebGPU) so you can skim the highlights & see links to resources mentioned by the speakers all in one place.

Event link where you can register for future meetups: https://www.khronos.org/events/webgl-webgpu-meetup-january-2022

General updates & new WebGL extensions

Ken Russell (https://twitter.com/gfxprogrammer)
Kai Ninomiya (https://twitter.com/kainino0x)
Slides

Ken encouraged everyone who works with or is interested in WebGPU to join the community! They welcome all feedback.

See the WebGPU GitHub discussions: https://github.com/gpuweb/gpuweb/discussions/
Also WebGPU matrix chat https://matrix.to/#/#WebGPU:matrix.org
This repo is where the spec is being developed & discussed

WebGPU spec 1.0 release is currently targeting ~Q2 2022.

You can release WebGPU apps on the web today without having users enable any special flags by signing your website up for the origin trial (https://web.dev/gpu/#register-for-ot).

There are currently many breaking changes as the spec rapidly develops. I was curious about these changes particularly with WGSL. Skimming the recent PRs you can find:

Adding increment/decrement operators . PR. Spec link.
Built in functions for converting to/from degrees & radians. PR. Spec link.

It is exciting for me to see that such core language features are still being added at this stage, and that we as developers get to provide feedback & be a part of this process!

WebGL 2 can now be considered universally supported!

This is largely thanks to WebGL 2 shipping on iOS/MacOS last year. It runs on Metal on all newer devices.
Safari team is rapidly fixing any WebGL 2 bugs. If you notice any please report it at https://bugs.webkit.org/.
Ken’s recommendation for most apps is that you no longer need to maintain a WebGL 1 fallback.

New WebGL 2 extensions

OES_draw_buffers_indexed — requested by 3D formats working group to implement advanced materials (eg using dual depth peeling). This allows you to use different blend options when writing to multiple color buffers simultaneously.
WEBGL_draw_instanced_base_vertex_base_instance — Reduces memory use by allowing reusing index buffer to draw multiple different geometries from the same set of vertex buffers. This is still a draft.
WEBGL_multi_draw_instanced_base_vertex_base_instance — Same as above but for rendering more than 1 primitive with one draw call.

WebGPU Best Practices

Brandon Jones (https://twitter.com/Tojiro)
Slides

Brandon has written a couple of WebGPU best practice guides, like this one about uploading data to GPU buffers: https://toji.github.io/webgpu-best-practices/buffer-uploads.html.

This presentation covered this and a few other other topics I found incredibly useful:

Use debug labels to get helpful error messages: https://gpuweb.github.io/gpuweb/explainer/#errors-errorscopes-labels
Use debug groups so that error messages will tell you exactly where in the pipeline the failure occurred (like which render pass): https://gpuweb.github.io/gpuweb/explainer/#command-encoding-debug
Use compressed textures (see artist’s guide here). Brandon built a library to serve as a reference for loading different texture formats in WebGPU/WebGL: https://github.com/toji/web-texture-tool.
gltf-transform has a CLI for creating these compressed textures & applying other optimizations: https://gltf-transform.donmccurdy.com/cli.html
Use async pipeline creation. Otherwise users will experience jank when you use a pipeline that’s not ready.
Avoid implicit pipeline layouts. Mainly so that you can re-use bind groups for different pipelines, which is the common scenario. For example you would have a bind group for your uniform buffer(s) that is re-used for all your shader pipelines.

WSGL Editor

Takahiro (https://twitter.com/superhoge)
Online demo: https://takahirox.github.io/online-wgsl-editor/index.html
Repo: https://github.com/takahirox/online-wgsl-editor

This is a ShaderToy-like environment but for WGSL & WebGPU. The editor will show you any errors inline.

It’s really useful for learning & experimenting. You can move your mouse away from the screen to hide the code & just see the display. Contributions are welcome!

Takahiro demonstrated using this editor to modify a vertex shader & deform this cube:

Creative coding with WebGL 2

Jaume Sanchez (https://twitter.com/thespite)
Online demo: https://spite.github.io/genuary-2022/
Repo: https://github.com/spite/genuary-2022

Jaume presented incredibly beautiful work he did for Genuary (https://genuary.art/) that’s all open source.

Jaume Sanchez Elias on Twitter: "#genuary 2022 - Day 21 - Combine two (or more) of your pieces from previous days to make a new piece.Day 4 + Day 16 = the DOFidenzamade with #threejs #genuary2022https://t.co/8RnqjSm0x3 pic.twitter.com/jUaYogaQGF / Twitter"

genuary 2022 - Day 21 - Combine two (or more) of your pieces from previous days to make a new piece.Day 4 + Day 16 = the DOFidenzamade with #threejs #genuary2022https://t.co/8RnqjSm0x3 pic.twitter.com/jUaYogaQGF

It’s a great resource for learning lots of different techniques for procedurally generated effects like generating noise, using SDF’s (signed distance fields), volume rendering, non photorealistic rendering, and more.

Clustered shading in PlayCanvas

Donovan Hutchence (https://twitter.com/slimbuck7)
Slides

Donovan presented on how the PlayCanvas team recently added clustered lighting to the engine. He gave a great explanation of how it works including some stunning debug views you can see in the video clip in the tweet below.

Will Eastcott on Twitter: "The @playcanvas engine repo was trending on GitHub last week. Maybe due to the clustered lighting tech we showed at @thekhronosgroup WebGL meetup. If u want to see how we did it:📜Repo: https://t.co/Tnf0ES1nDZ🖥️Slides: https://t.co/zJ7s4XWjpO📺Preso: https://t.co/H2F71X0hp9 pic.twitter.com/rfMSHAUewh / Twitter"

The @playcanvas engine repo was trending on GitHub last week. Maybe due to the clustered lighting tech we showed at @thekhronosgroup WebGL meetup. If u want to see how we did it:📜Repo: https://t.co/Tnf0ES1nDZ🖥️Slides: https://t.co/zJ7s4XWjpO📺Preso: https://t.co/H2F71X0hp9 pic.twitter.com/rfMSHAUewh

I hope you found this useful! You can find me on Twitter (https://twitter.com/Omar4ur), I’d love to hear any thoughts.

If you’re interested in giving a talk at this meetup, reach out to them! From their event description:

If you would like to be considered to present at one of our upcoming WebGL+ WebGPU Meetups, please reach out to events@khronos.org.

My Takeaways from WebGPU July 2021 Meetup

Omar Shehata — Tue, 24 Aug 2021 19:34:49 GMT

The last WebGL / WebGPU meetup was packed with a lot of really exciting content. You can find the full 90 minute recording here. I decided to put together my takeaways here for others who want a quick glimpse of the latest updates, focusing particularly on WebGPU.

You’ll find Youtube timestamps for each talk below if you’re interested in watching the full segment (which I highly recommend!)

Advice for porting a large WebGL project to WebGPU

Jasper St. Pierre — Yacht Club Games.
Timestamp: 6:02–21:25
Slides

Jasper is the creator of the famous noclip.website — a custom web rendering implementation of many famous video game levels, like Mario Kart & Half Life.

He recently ported this engine to WebGPU. This acted as a really good testbed for the spec because his engine implements so many different games that all use a wide range of rendering techniques.

WebGPU requires the use of uniform buffers, which is one big difference with WebGL. You can’t define and update individual uniform values anymore. You put all your uniforms into a buffer and send that all at once, which is a more efficient use of modern GPU hardware.
Jasper recommends porting WebGL 1 -> WebGL 2 as an intermediate step. Since WebGL 2 supports uniform buffers, and the code for binding is a little simpler.
Do not update uniform buffers between draw calls in WebGPU. This may be a common pattern in WebGL 2, but you want to keep GPU memory untouched as long as possible so it can execute draw calls in parallel. Instead, he created one single large buffer, containing data for all draw calls, and binding different parts of it before different draw calls.

Jasper kept his shaders in GLSL and converts to WGSL at runtime using https://github.com/gfx-rs/naga using WebAssembly. WGSL stands for WebGPU Shading Language.
Viewport and clipspace conventions are different in WebGPU. WebGPU has 0,0 in the top left (as opposed to bottom left). And the frustum in WebGPU is 1 to 0 (as opposed to 1 to -1). This was easy to account for.
Consider a “draw call objects” design for your renderer. This one is more general advice: Instead of each object initiating draw calls directly, it pushes an object into a list containing info needed for the draw call. This makes it easier to handle multi pass rendering, sorting if needed for transparency, and collecting uniform data up front.

No.clip website repo: https://github.com/magcius/noclip.website

The graphics framework behind it: https://github.com/magcius/gfxrlz

glTF Sampler Viewer

Moritz Becher — UX3D
Timestamp: 21:35–35:30
Slides

glTF Sample Viewer (https://github.khronos.org/glTF-Sample-Viewer-Release/) is an open source web app that‘s really useful for (1) verifying your glTF implementation is correct (2) finding clean sample implementation of any part of the glTF spec (3) easily previewing glTF models by drag & drop.

Source code: https://github.com/KhronosGroup/glTF-Sample-Viewer

This is a “native glTF viewer” in that it uploads glTF resources/buffers directly to the GPU, as opposed to parsing them into another intermediate format. .
Always updated with the latest spec & extensions. For example, here you can find an implementation of the new volume transmission extension, which is useful since the spec doesn’t dictate a specific implementation.

PBR & KTX extensions for glTF

Sandra Voelker — Target
Timestamp: 35:40–54:20
Slides

This talk was really interesting to see how a lot of the PBR (Physically Based Rendering) extensions for glTF are being pushed for by companies like Target & Wayfair for eCommerce purposes.

Some of those new extensions this year are:

KHR_materials_sheen — for representing the back-scattering of velvet-like materials (cloth & fabric generally).
KHR_materials_specular — for representing specular color refraction (like how different rainbow-like colors can appear when looking through glass).
KHR_materials_transmission & KHR_materials_volume — for translucent objects with non-uniform thickness like a glass jar.
KHR_texture_basisu — for texture compression with Basis Universal supercompression.

Since these extensions are new, not all 3D authoring programs support exporting them yet. Some tools Sandra recommended are:

BabylonJS exporter for 3DS Max and Maya. This is a plugin that lets you export a glTF that uses these new PBR extensions
Sandra used Blender for exporting the vase with transparent glass. See Blender glTF extensions doc.
Sandra used the new Adobe Substance 3D to export cloth materials with the sheen extension.
Sandra used the glTF VS Code plugin for adding the clear coat extension. This allows you to edit glTF material properties in realtime by editing the JSON file directly.

There are many useful links at the end of Sandra’s slides, I wanted to call out in particular this “artist’s guide” for creating KTX materials which is really helpful for understanding how to get textures that use vastly less GPU memory and still look great: https://github.com/KhronosGroup/3D-Formats-Guidelines/blob/main/KTXArtistGuide.md.

Gestaltor (https://gestaltor.io/) is also an awesome glTF editor made by the same people that made the glTF viewer in the previous talk.

Using multi-draw to speed up drawing many small objects

Philip Taylor — Zea
Timestamp: 54:40–1:12:20
Slides

This talk was about using a new WebGL extension (WEBGL_multi_draw) to efficiently draw millions of small objects, which are common for 3D design files.

Normally you might need to use many draw calls to render all these different objects because they have different textures/shaders/material properties. For these scenes where number of draw calls is the bottleneck, Philip described a technique to bind all data required ahead of time, and group these hundreds of thousands of draw calls into much fewer “multi draw” calls.

“If instancing is drawing the same geometry many times, multi draw is drawing different geometries many times” is how Philip described it. So it’s getting the same performance benefit of instancing with more flexibility.
He referenced this blog post as a good source to learn about it: The road to 1 million draws
You essentially pack all your geometry into one huge vertex buffer which you then pass to the multi-draw call with offsets that define geometry for separate objects
You have a “drawId” which can be used in the shader to figure out which object is being drawn in the multi-draw call
It is not possible to switch/bind different uniforms across different objects, since they are all grouped into one multi-draw call
So they bind everything into textures, including model matrices. The drawId is used to index into these textures and get the model matrix, material properties, etc.
The disadvantage of this is that all these properties have to be the same precision, so this will be wasting extra memory. This is also a very non-trivial architecture, and makes it very difficult to change materials at runtime.

It was really cool to see at the end that there was a huge performance boost even when the multi-draw extension wasn’t supported. The fallback is to issue individual draw calls, but not having to re-bind textures/uniforms between draw calls was a big boost. They showed a design model with 17k parts, 2.5 million triangles, 17k draw calls, running at 36 fps on an iPad in Safari

Speeding up Unity’s WebGL export with batching techniques

Brendan Duncan — Unity
Timestamp: 1:13:00–1:30:11
Slides

Brendan shared how they were able to significantly speed up Unity’s WebGL export when drawing many different objects with unique materials. Normally you can only batch objects that share the same material/shader.

Brendan presented a technique to batch objects that have different materials by binding all data ahead of time and configuring each shader to use the right data (Unity calls this the “SRP Batcher”).

They initially couldn’t use this technique as-is in WebGL because it required 2 features not supported in WebGL: uniform layout locations and buffer binding points. This is what allows the shaders to reference the right data when it is all sent to the GPU ahead of time.
Specifically the issue is that WebGL does not use integer uniform locations. This is done as a security/sandboxing measure
They worked around this by creating an open source Emscripten plugin: https://github.com/emscripten-core/emscripten/pull/13887
This plugin preprocesses the shaders. You can write OpenGL shaders using this feature (so they can keep the shaders as-is for Unity desktop/other platforms), the plugin will create a dictionary of all the uniform binding points, and generate GLSL code that works in WebGL with the correct bindings.

The result is a 2x improvement in the worst case scenario, for a scene with 1,600 objects, each with a different material, 4 real-time lights and 1 directional shadowmap.

I hope you found this useful! You can find me on Twitter (https://twitter.com/Omar4ur), I’d love to hear any thoughts.

You can sign up to be notified of the future WebGL/WebGPU meetups here: https://www.khronos.org/news/subscribe/. You can also participate in the WebGL developers mailing list which is an active community of people sharing awesome projects, exchanging feedback, and asking questions.

Create a Complete Satellite Tracker from Scratch in JavaScript

Omar Shehata — Fri, 07 May 2021 23:18:46 GMT

This tutorial will walk you through how to create a web app that visualizes the location of any satellite in real-time, like the International Space Station.

We’re going to do this from scratch, using the same techniques a real rocket scientist would!

We’ll look at where to get the data about individual satellites that the government publishes, known as Two-Line Element Sets, or TLE’s.
We’ll use satellite-js to predict the orbit of the satellite given the TLE’s (this is the rocket science part).
We’ll use CesiumJS to visualize the result, but you can use any library/engine that can take in longitude, latitude, and height.

Below is a preview of the final result.

https://medium.com/media/e20c405b6909206ed6143e22f997f3ae/href

This is showing the path of the International Space Station, sped up by 40x. To see its current location in real-time, click the clock icon at the top-left of the clock wheel.

Here’s a direct link to the app. And the source code on Glitch.

1 — Get the satellite’s Two-Line Element Set

A Two-Line Element Set, or TLE, is a data format that describes the motion of an object orbiting the Earth. It was created by the North American Aerospace Defense Command (NORAD). You can read more about it and its history here.

Given this description of the orbit, we can predict the location of where it’s going to be at any moment in time (which is step 2 below).

This means that most “live” satellite trackers are not live in the same way that tracking a delivery car on a map is. Instead of relying on constantly receiving position updates, those who track objects in space will often get the latest TLE’s (which are regularly updated) and use that to predict where the object is right now.

Where do we get the TLE’s? There is no one global official registry. Whoever owns the satellite and is monitoring it is responsible for updating and publishing the TLE for the benefit of the global space community (unless it’s a spy satellite).

We can find these TLE’s on Space Track which is a registry run by the United States Space Command.

Another source is this list on CeleStrak maintained by Dr. T.S. Kelso.

We’re going to use CeleStrak since it doesn’t require a login. To find the TLE for the International Space Station, click on the Space Stations link.

The first one is the TLE for the ISS:

ISS (ZARYA)             
1 25544U 98067A   21122.75616700  .00027980  00000-0  51432-3 0  9994
2 25544  51.6442 207.4449 0002769 310.1189 193.6568 15.48993527281553

The meaning of these numbers is listed in table 1 in Dr T.S. Kelso’s column. Most of them are identifiers and metadata about the satellite, like when it was launched.

You can find TLE’s for weather satellites, GPS satellites, and even SpaceX’s Starlink constellation in this same format.

2 — Predict the satellite orbit

Now that you know how to get the TLE of the object you’re interested in tracking, the next step is converting that to a position in time.

We’re going to use satellite-js for this.

Include the library from a CDN:

https://medium.com/media/2f3a529b762b8a3c74cad54aa59887c8/href

Then pass the TLE to it, and a time:

https://medium.com/media/6137569e24d81eb2e9f665298974666c/href

Now we have the position of the satellite at the current time, new Date().

This position is produced as a result of simulating a specific model of satellite motion. This model is called SGP4/SDP4. All TLE’s assume this specific model.

If you’re wondering about the accuracy of this model, the short answer is, it depends.

Accuracy of the two-line element sets is dependent upon a number of factors. These range from the particular sensors used and amount of data collected to the type of orbit and condition of the space environment. Unfortunately, since these factors vary for each element set, so does the accuracy. While NORAD has experimented with methods to incorporate prediction quality into the element sets, none of these methods has yet proved successful.

3 — Visualize the result

Now we have a way to get the location of any satellite, at any given time. We can pass in future times to animate its path, which we’ll do in the next step.

First, let’s see how to visualize an individual point in space using CesiumJS.

We load the library from CDN:

https://medium.com/media/2c4527c4f2254a0d37061e3aa6a6768b/href

And create a container element:

https://medium.com/media/782d07c40cad14b786795f6efc04d71c/href

We then initialize the viewer. Here we pass in some extra options to disable functionality that requires an access token:

https://medium.com/media/cda9f9e7cc1501ff02506e415487592c/href

Finally, we’ll visualize the satellite position as a red dot in space:

https://medium.com/media/7007a794ecca7681ef6c12850ae59a97/href

See the complete source code of this step in simple.html on Glitch.

4 — Animate the path

To animate the path, we just need to sample more positions in the future. CesiumJS has a built-in way to interpolate between these samples over time.

The setup for this is a bit verbose. You can see the full code on Glitch. The important concepts are described below.

We create a SampledPositionProperty. This is an object that will hold position samples over time and will interpolate between them:

https://medium.com/media/13e0156e2dcc1737fa77ac138e4b2ed7/href

We loop through however many samples we want to get, and for each sample, we construct a time object, called JulianDate in CesiumJS, and a position, and we add that as a sample:

https://medium.com/media/2332306ab023506f652dbb7477218fc5/href

Finally, we pass this positionsOverTime object to our point.

https://medium.com/media/40b50d89de948e96f6398ed710ed1cf5/href

The point will move as the timeline at the bottom moves. To attach the camera to the moving point we do:

https://medium.com/media/9f1c4f9a46376bb13b2b9944daf4d3a5/href

Conclusion

I hope you enjoyed learning a little bit about what goes into building a satellite tracker. There’s a lot more to the topic we didn’t touch on, like what exactly do the parameters in the TLE mean? How often are they updated? How are they updated?

I don’t know, but I find it really empowering to know what formats this kind of data is published in & where to get it, and quite amazing that we can do all this directly in the browser with JavaScript!

Here are a couple of fun ideas to explore now that we can do this:

Visualize multiple satellites, like the entire Starlink constellation. Inspired by Celestrak’s viewer which shows every satellite in its catalog. Perhaps visualize how the number of Starlink satellites grew over time?

Or simulate what it would look like from street level. Maybe add buildings/elevation data to find the best place in the city to see the satellite?

There’s a prototype of this in street-level.html in the Glitch source code. Demo: https://satellite-viewer.glitch.me/street-level.html.

See also James Darpinian’s “See a satellite tonight” app which uses a combination of CesiumJS and Google street view.

It might also be fun to use a 3D model of the right scale instead of a dot, and get a real sense of how close satellites get to each other in space.

Thanks for reading! If you found this helpful, follow me on Twitter @Omar4ur to see more of my work. Otherwise, you can also find me here.

More content at plainenglish.io

Create a Complete Satellite Tracker from Scratch in JavaScript was originally published in JavaScript in Plain English on Medium, where people are continuing the conversation by highlighting and responding to this story.

How to Load Sketchfab Models Directly in a Three.js App

Omar Shehata — Tue, 27 Apr 2021 23:55:16 GMT

Photo by Christopher Gower on Unsplash

Sketchfab’s API gives you programmatic access to the largest collection of glTF 3D models on the web. This article walks you through a minimal code example to show you how to:

Let your users authenticate with the Sketchfab API
Download a 3D model as a zip file containing the glTF
Load this zip file into Three.js

Source code:

OmarShehata/threejs-sketchfab-example

How it works

I originally implemented this to let readers of my WebGL outlines tutorial see how the effect looked on test cases of their choosing since I kept finding algorithms that didn’t work on my specific corner cases (but I wouldn’t find out until after I implemented it/downloaded and ran it).

It’s a really easy way to let users bring in their own data (or millions of example models).

You can see how this works in this outlines live demo:

Open the demo: https://omarshehata.github.io/csb-l01dp/
Click Login to Sketchfab
When directed back to the app, paste a link of a model in the Sketchfab URL field, like this: https://sketchfab.com/3d-models/skull-downloadable-1a9db900738d44298b0bc59f68123393

The model must be downloadable by the Sketchfab account that’s logged in.

The demo app loads a 3D model directly from any Sketchfab URL into a ThreeJS scene

1 — Authenticate with the Sketchfab API

The first step is to register your app with Sketchfab. The instructions for this are here:

OAuth Login - Developers

The current process at the time of writing is to contact them.

You’ll need to pick a redirect URI. This should be the final URI where you will deploy your app.

You’ll want to use the Implicit grant type. We can’t keep an API key a secret in a web app, so we instead rely on the redirect URI (if someone malicious uses your client ID, Sketchfab will redirect them to your real app after login, regardless of who may have initiated the login).

After your register your app you’ll have a Client ID.

You’ll use this to send the user to Sketchfab for login as shown here:

const CLIENT_ID = 'YOUR_CLIENT_ID_HERE';
const AUTHENTICATION_URL = `https://sketchfab.com/oauth2/authorize/?state=123456789&response_type=token&client_id=${CLIENT_ID}`;
window.open(AUTHENTICATION_URL, '_blank');

Once the login is done, you’ll need to grab the access token from the URL. You can find a code snippet in the source code that does this and stores it in local storage here:

https://github.com/OmarShehata/threejs-sketchfab-example/blob/b780b8c1eac37fc71d8d803fc465c874efd5954c/SketchfabIntegration.js#L77-L92

You’ll use this token for subsequent API calls.

2 — Download the 3D model

Once you have a token, you can use this to fetch a download URI for the glTF model.

Here is the code snippet for fetching the download URI given any Sketchfab URI like this: https://sketchfab.com/3d-models/skull-downloadable-1a9db900738d44298b0bc59f68123393

https://github.com/OmarShehata/threejs-sketchfab-example/blob/b780b8c1eac37fc71d8d803fc465c874efd5954c/SketchfabIntegration.js#L95-L115

Note that this request will fail if the user who is logged in does not have access to download this model. This will be true for store models that require a purchase, or free models that are not downloadable.

You can filter for downloadable models in the Sketchfab search:

This will show you either free models you can download or models that can be purchased.

Now that we have a URL to a zip file containing a glTF model, we can pass that to ThreeJS to load it.

3 — Load the ZIP file into ThreeJS

This was the tricky part of me. Normally ThreeJS requires a direct URL to the glTF file. To load it from a zip file we’re going to (1) unzip the contents into memory using JSZip.

We can’t just pass the unzipped raw content to ThreeJS, because a glTF file may reference other files by a filepath (image or geometry resources). So we need to (2) create a Blob for each resource and (3) override the resources the glTF file is requesting with the Blob URI’s.

For example, if the glTF file is trying to load “textures/defaultMat_baseColor.jpeg” as a relative filepath, we detect this, and instead pass this URI:

//Unzip from JSZip
const file = unzippedBaseColorJPEGFile;
// Create a Blob from this file
const blob = await file.async('blob');
// Create a URL to this Blob
const baseColorBlobUrl = URL.createObjectURL(blob);
// Use ThreeJS's loading manager so instead of loading from relative filepaths we load from the blob URI's we created

const loadingManager = new THREE.LoadingManager();                            loadingManager.setURLModifier((url) => {

  if (url == 'textures/defaultMat_baseColor.jpeg') {
    return baseColorBlobUrl;
  }

});

We use ThreeJS’s LoadingManager to do this.

Here is the code snippet that can take a URL to any zipped glTF and load it in ThreeJS:

https://github.com/OmarShehata/threejs-sketchfab-example/blob/b780b8c1eac37fc71d8d803fc465c874efd5954c/SketchfabIntegration.js#L30-L70

4 — Display attribution

Using 3D models from Sketchfab in your application requires that you display attribution to the original author. Read more about this on Sketchfab.

We can automatically get the attribution & license information from this Data API route.

Here is a function that will construct the attribution text given a modelID:

https://github.com/OmarShehata/threejs-sketchfab-example/blob/8d126d00faf47fb0b99f9f2f7c78a25332ccfb7b/SketchfabIntegration.js#L121-L142

The example app will display the attribution in the bottom left corner, linking to the original model URL, author’s Sketchfab profile, and the license.

Known issues

One problem with loading some Sketchfab models is that their scale will be much bigger than the current viewport. Another problem is some models may not be centered around the origin, so they may not be visible when loaded.

Normalizing and scaling models when loading them in ThreeJS would help solve this, similar to how Don McCurdy’s glTF Viewer works.

Thanks for reading! If you found this helpful, follow me on Twitter @Omar4ur to see more of my work. Otherwise, you can also find me here.

More content at plainenglish.io

How to Load Sketchfab Models Directly in a Three.js App was originally published in JavaScript in Plain English on Medium, where people are continuing the conversation by highlighting and responding to this story.

How to render outlines in WebGL

Omar Shehata — Mon, 18 Jan 2021 21:25:17 GMT

This article describes how to visualize outlines for a WebGL scene as a post process, with example implementations for ThreeJS & PlayCanvas.

Left — boundary outline only. Right — the technique described in this article. Boat model by Google Poly.

Note: I’ve written a follow up that builds on this article with an improved technique that solves many of the artifacts present here. See Better outline rendering using surface IDs with WebGL.

There are a few common approaches that produce boundary-only outlines as shown on the left of the above picture.

Drawing objects twice, such that the backfaces make up the outline, described here.
A post process using the depth buffer, implemented in ThreeJS here.
Similar post process implemented in PlayCanvas here.

Rendering the full outlines of a scene is particularly useful when you need to clearly see the geometry and structure of your scene. For example, the stylized aesthetic of Return of the Obra Dinn would be very hard to navigate without clear outlines.

Top, a stylized two-tone lighting in ThreeJS inspired by Return of the Obra Dinn. Bottom, the same scene with outlines. Ship model from Museovirasto Museiverket Finnish Heritage Agency on Sketchfab.

The technique I describe here is similar to the post process shaders linked above, with the addition of a “normal buffer” in the outline pass that is used to find those inner edges.

Live Demo

Below is a link to a live demo of this technique implemented in ThreeJS. You can drag and drop any glTF model (as a single .glb/glTF file) to see the outline effect on your own test models:

https://threejs-outlines-postprocess.glitch.me/

You can also find the source code on GitHub: https://github.com/OmarShehata/webgl-outlines.

Overview of the technique

Our outline shader needs 3 inputs:

The depth buffer
The normal buffer
The color buffer (the original scene)

Given these 3 inputs we will compute the difference between the current pixel’s depth value and its neighbors. A large depth difference tells us there’s a distance gap (this will typically give you the outer boundary of an object but not fine details on its surface).

We will do the same with the normal buffer. A difference in normal direction means a sharp corner. This is what gives us the finer details.

We then combine those differences to form the final outline, and combine that with the color buffer to add the outlines to the scene.

Tip: The live demo has a scaling factor for each of the normal & depth. You can scale that to 0 to see the influence of each on the final set of outlines.

Overview of the rendering pipeline

Here is how we’re going to set up our effect:

This effect requires 3 passes. Two render-passes and one post-process.

Render pass 1 captures the color of all objects in the scene in “Scene Buffer”.
It also outputs the depth of every pixel in a separate “Depth Buffer”.

Render pass 2 re-renders all objects in the scene with a normal material that colors it using the object’s view-normal at every pixel. This is written
to the “Normal Buffer”.

Finally, Outline pass is a post process, taking the 3 buffers and rendering onto a fullscreen quad.

Each stage in this rendering pipeline visualized.

This can be further optimized by modifying the engine to combine the normal and depth buffers into one “NormalDepth”, similar to how Unity does it, to avoid the need for the 2nd render pass.

A final step not shown in the diagram is an FXAA pass, which we need because we’re rendering the scene onto an off-screen buffer, which disables the browser’s native antialiasing.

Implementation

It’s difficult to describe this technique without reference to a specific engine since a core part of it is how to set up the rendering pipeline described above. The implementation details here will be specific to ThreeJS but you can see the PlayCanvas source code along with an editor project here:

https://github.com/OmarShehata/webgl-outlines/tree/main/playcanvas

1. Get the depth buffer

3D engines will typically draw all opaque objects into a depth buffer to ensure objects are rendered correctly without having to sort them back to front. All we have to do is get a reference to this buffer to pass it to our outline post process.

In ThreeJS, this means setting depthBuffer = true on the render target we’re creating so that we capture the “scene color” and the “depth buffer” at the same time. See: https://threejs.org/docs/#api/en/renderers/WebGLRenderTarget.depthBuffer

In our demo this is created here:

https://github.com/OmarShehata/webgl-outlines/blob/4d94f360e083540c8bb860bed3509c0d4528d22c/threejs/src/index.js#L31-L42

There are a few caveats to know when working with the depth buffer:

You need to know how the values are “packed”. Given the limited precision, does the engine just linearly interpolate Z values camera.near to camera.far? Does it do this in reverse? Or use a logarithmic depth buffer?
The engine most likely already has some mechanisms for working with depth values that you can re-use. For ThreeJS, you can include #include <packing> in your fragment shader which will allow you to use these helper functions.
For just visualizing it for debug purposes, you can collapse your camera’s near/far to cover the bounds of the object so you can more clearly see the image.

2. Create a normal buffer

If your engine supports outputting the normals of everything in the scene, you should use that directly. Otherwise, you’ll need to create a second render pass. This needs to be identical to the original render, with the only exception that all materials on all meshes are replaced by a “normal material” that renders the view space normals.

ThreeJS has a convenient scene.overrideMaterial method we can use for exactly this purpose. Instead of creating a new identical scene and a new identical camera, we can directly re-render the same scene with the given override material.

this.renderScene.overrideMaterial = new THREE.MeshNormalMaterial();

renderer.render(this.renderScene, this.renderCamera);

this.renderScene.overrideMaterial = null;

In our ThreeJS implementation this is encapsulated in CustomOutlinePass.js for convenience, but it is a completely separate render pass.

3. Create the outline post process

The outline effect is a post process — we’ve already rendered the scene, now we need to take those buffers, combine them, and render the result onto a fullscreen quad. The result of that will either go directly to the screen or to the next pass in the pipeline (like FXAA).

We need to pass 3 uniforms: sceneBuffer, depthBuffer, and normalBuffer.

We create helper functions to read the depth at an offset from a given pixel. Then we sum up the difference between the current pixel’s depth value and its neighbors.

float depth = getPixelDepth(0, 0);                           
// Difference between depth of neighboring pixels and current.                           
float depthDiff = 0.0;                           
depthDiff += abs(depth - getPixelDepth(1, 0));                           depthDiff += abs(depth - getPixelDepth(-1, 0));                           depthDiff += abs(depth - getPixelDepth(0, 1));                           depthDiff += abs(depth - getPixelDepth(0, -1));

The same thing is done for normals as well. Since the normal is a 3 dimensional vector, we get the difference using the distance function.

vec3 normal = getPixelNormal(0, 0);
// Difference between normals of neighboring pixels and current                           float normalDiff = 0.0;                           
normalDiff += distance(normal, getPixelNormal(1, 0));                           normalDiff += distance(normal, getPixelNormal(0, 1));                           normalDiff += distance(normal, getPixelNormal(0, 1));                           normalDiff += distance(normal, getPixelNormal(0, -1));

To render the outline only at this point we would do:

float outline = normalDiff + depthDiff;
gl_FragColor = vec4(vec3(outline), 1.0);

There’s a few parameters here to tweak:

We can include the diagonals in our neighbor sampling to get a more accurate outline
We can sample one or more neighbors further, to get thicker outlines
We can multiply normalDiff and depthDiff by a scalar to control their influence on the final outline
We can tweak normalDiff and depthDiff so that only really stark differences in depth or normal direction show up as an outline. This is what the “normal bias” and the “depth bias” parameters control.

This is implemented in CustomOutlinePass.js.

4. Combine the outlines with your final scene

Finally, to combine the outline onto the scene, we mix the scene color with a chosen “outline color”, based on our outline value.

float outline = normalDiff + depthDiff;
vec4 outlineColor = vec4(1.0, 1.0, 1.0, 1.0);//white outline
gl_FragColor = vec4(mix(sceneColor, outlineColor, outline));

This is also where you can create any custom logic for how you combine your outline with your scene.

For example, in the Return of the Obra Dinn, the outlines change color based on the lighting. To achieve this effect we would check the lighting direction against the surface normal in our normal buffer, and color the outline white if it not in direct light, and black if it is facing the light source(s).

Stylized lighting in ThreeJS inspired by Return of the Obra Dinn. Notice that the outlines change color based on the scene’s lighting.

Thanks for reading! If you found this helpful, sign up to my newsletter to follow my work & stay in touch.

Thanks to Ronja Böhringer whose Outlines via Postprocessing article helped me understand this technique and adapt it for the web.

How to Fix Gimbal Lock in N-Dimensions

Omar Shehata — Sun, 23 Aug 2020 04:12:11 GMT

Gimbal lock is a common issue that arises in 3D rotation systems. Conventional wisdom says that you should represent your rotations as quaternions to avoid this problem (as stated in many popular game engine docs and countless YouTube videos). But I couldn’t find any good explanations of exactly how quaternions solve it.

This was important for me because I was implementing a 4D geometry viewer and spent a lot of time trying to figure out how to generalize quaternions to higher dimensions, only to discover that you can still get gimbal lock with quaternions!

The answer is it’s not about what you use to represent your rotations (quaternions, matrices, or rotors), it’s about how you apply those rotations.

What causes gimbal lock?

Gimbal lock is the loss of a degree of freedom in a rotation system. It will always happen in any system that uses Euler angles, where you rotate your object by applying a fixed set of successive rotations.

In 3D, this happens when one of the 3 axes of rotation becomes parallel to any other one (or in other words, when the “gimbals” line up). This is shown in the animation below. It starts with the +Z side facing the camera. The first 90 degree rotation places the top cube in a gimbal locked state: rotating by the blue and red gimbals now produce the same rotation. It is not possible in this state to rotate around the cube’s local X axis (pointing towards the camera). This is the degree of freedom we’ve lost and is represented by the red gimbal in the bottom view.

It’s easier to see this by trying it yourself in this demo (requires a keyboard):

Rotate the cubes with W,A,S,D and Q, E
R to reset

https://gimbal-lock-quaternions.glitch.me/

The top two cubes use Euler angles, so there will always be a way to get into this gimbal locked state. All you have to do is line up the blue and red gimbals. Then compare the behavior with the bottom two cubes, which show what the correct rotation should be.

Here is pseudocode for how the rotation for the top left cube is implemented:

rotationAroundX = Matrix3.fromAxisAngle(angle1, Xaxis);
rotationAroundY = Matrix3.fromAxisAngle(angle2, Yaxis);
rotationAroundZ = Matrix3.fromAxisAngle(angle3, Zaxis);

cubeRotation = rotationAroundX * rotationAroundY * rotationAroundZ;

This is an Euler angle system that uses rotation matrices to represent and apply the rotations. We will get exactly the same behavior if we swap out the matrices for quaternions.

Here is how the top right cube is implemented:

rotationAroundX = Quaternion.fromAxisAngle(angle1, Xaxis);
rotationAroundY = Quaternion.fromAxisAngle(angle2, Yaxis);
rotationAroundZ = Quaternion.fromAxisAngle(angle3, Zaxis);

cubeRotation = rotationAroundX * rotationAroundY * rotationAroundZ;

This is an Euler angle system that uses quaternions to represent and apply the rotations, and thus also suffers from gimbal lock.

How do you fix gimbal lock?

To fix gimbal lock, we must avoid modelling this physical gimbal system.

In 3D, instead of using 3 fixed angles that we multiply together to get the final rotation, we will:

Construct a quaternion that describes a rotation around whatever axis we want, and the angle to rotate by.
Multiply that by the object’s current rotation as represented by a quaternion.
Take the result and overwrite it back into the object’s current stored rotation.

This is how the bottom right cube in the demo above is implemented. The pseudocode looks like this:

// We want to rotate by 0.05 radians in this frame.
rotationThisFrame = Quaternion.fromAxisAngle(0.05, arbitraryAxis);

cubeRotation = cubeRotation * rotationThisFrame;

Nothing about this technique is unique to quaternions. Here is how the bottom left cube implements this with matrices:

rotationThisFrame = Matrix3.fromAxisAngle(0.05, arbitraryAxis);

cubeRotation = cubeRotation * rotationThisFrame;

The “arbitrary axis” is the cube’s local X/Y/Z axes, depending on what key is pressed. It can instead be the global X/Y/Z axes if we wanted to rotate relative to a fixed global frame.

You can see the full JavaScript code for all 4 rotating cubes here: https://glitch.com/edit/#!/gimbal-lock-quaternions?path=index.html%3A51%3A37

Generalizing to N-dimensions

To generalize this solution to higher dimensions we need two things:

A way to represent rotations. So far in 3D we’ve used 3x3 matrices and quaternions.
A way to describe a rotation around an arbitrary axis.

Side note: it’s more generalizable to talk about “rotating within a plane” instead of “rotating around an axis”. In 2D there is only one plane of rotation, the XY plane. In 3D there are 3 planes of rotation. In 4D there are 6 planes. In N-D there are (N choose 2) planes.

I ended up choosing 4x4 matrices to represent my rotations, because 4x4 matrix operations are supported on the GPU and so I could rotate the geometry in my vertex shader, in the same way I would in 3D.

So the proposed solution is:

rotationThisFrame = Matrix4.fromAxisAngle(0.05, arbitraryAxis);

cubeRotation = cubeRotation * rotationThisFrame;

Now the tricky part is implementing a Matrix4.fromAxisAngle function. It’s much easier to create a 4x4 rotation matrix for each of the 6 planes of rotations and compose them. So this can be broken down into:

// Check which key(s) were pressed. In this frame, only the key for rotating in the XZ plane was pressed, so the rXZ 4x4 matrix gets a non-zero angle.

rXY = Matrix4.fromXYRotation(0);
rXZ = Matrix4.fromXZRotation(0.05);
rYZ = Matrix4.fromYZRotation(0);
rXW = Matrix4.fromXWRotation(0);
rYW = Matrix4.fromYWRotation(0);
rZW = Matrix4.fromZWRotation(0);

rotationThisFrame = rXY * rXZ * rYZ * XW * rYW * rZW;

cubeRotation = cubeRotation * rotationThisFrame;

This looks suspiciously like Euler angles, but it’s not. This will never gimbal lock. No matter what orientation the cube ends up in, we can always rotate it by 0.05 radians in any of the 6 planes.

To illustrate the difference, here’s a version that will gimbal lock:

// Check which key(s) were pressed. In this frame, only the key for the rotating in the XZ plane was pressed, so we add 0.05 to angle2.

angle2 += 0.05;

rXY = Matrix4.fromXYRotation(angle1);
rXZ = Matrix4.fromXZRotation(angle2);
rYZ = Matrix4.fromYZRotation(angle3);
rXW = Matrix4.fromXWRotation(angle4);
rYW = Matrix4.fromYWRotation(angle5);
rZW = Matrix4.fromZWRotation(angle6);

cubeRotation = rXY * rXZ * rYZ * XW * rYW * rZW;

Beyond 4D, you could continue using NxN rotation matrices, but since GPUs don’t support that, I’d use rotors from Geometric Algebra to simplify implementation, which is how you generalize quaternions to higher dimensions.

I hope you’ve found this post useful in illustrating how to avoid gimbal lock, regardless of how you choose to represent your rotations.

If you want to learn more about my 4D geometry work, check out my talk at !!Con. If you’ve found an error in my explanation or something is unclear, you can find me on Twitter (https://twitter.com/omar4ur).

Handmade GitHub Pages

Omar — Wed, 07 Jan 2026 14:50:03 +0000

This is part 2 of the "handmade computational sandbox" class. In part 1 (https://dev.to/omar4ur/handmade-cellular-automata-1f3c) we learned how to run a piece of JS code in a single HTML file on your computer, and how to deploy it on CodePen to share.

Today we're going to use GitHub for free static hosting, instead of CodePen. CodePen is good for quick experiments, GitHub is good for long term hosting.

For example, if you run a local community and want to set up a web page for it, a simple HTML page + a GitHub repo is perfect because (1) it's 100% free (2) if one day you leave town, the source code is available so someone else can copy it and continue to host it. This is designed to be very "permacomputing" and in line with Derek Siver's tech independence.

Example of a local community website made this way (just a single HTML file): https://mplawley.github.io/tundra-tech-talks/

If you're sharing a research prototype, this is also the perfect way to do it (live demo + source code available + a space for people to ask questions on the GitHub repo)

We're going to take the HTML/JS cellular automata simulation we created in part 1 and host it on GitHub pages. If you don't have your file from part 1, you can use this one that implements the game of life rules:

https://gist.githubusercontent.com/OmarShehata/9215085ff48882d1756ddd5ce2e0b912/raw/6ec921b6abd3dfc9f52a9be45989334cb1bb3c15/game_of_life.html

1. Create a new GitHub repo

Login or create an account on https://github.com/
Click "+" button in the top right -> "New repository"
Give it a name, toggle the "Add README" button. Click "create repository"

2. Add your HTML file

Click the "+" button in the top right of the repo and "create new file"

Call it index.html, and paste in your HTML/JS code

"index.html" is a special name, web servers will default to serving this as the homepage if they find it in a directory.

Click "commit changes"

3. Tell GitHub to deploy this website

Go to "settings" in the top right

Then "pages" in the left side panel

Here you will find the default settings for GitHub Pages, "deploy from branch". Just select your main branch here:

For bigger projects, you will have the code be in one branch, and the "built" thing to deploy in a different branch. For now, the code we are writing is itself the final code that runs in the browser, so we don't need this build step ("GitHub Actions" can be configured to run these build steps for you)

Hit "save"

4. Check the deployment status

If it worked, you should see this yellow circle back in the main repo page:

You can click on it to check the status. Once it's done, you can get the URL for your deployed website back in the Settings > Pages

Handmade Cellular Automata

Omar — Tue, 06 Jan 2026 18:59:09 +0000

This tutorial will walk you through creating & deploying your first "computational sandbox", an environment where you can define the rules of a simulation and explore the result. This is designed to be walked through together with a mentor.

This is the opposite of vibe coding - in order for you to use these simulations to think & learn about the world, you need to know everything about how it works. This is why we're doing everything "by hand". We're going to do the absolute minimum setup you need to (1) run things on a computer by yourself (2) share it with collaborators in a way that they can edit and build on.

Inspirations

Permacomputing
Derek Sivers' "tech independence" philosophy
- See also Rosano's Zero Data Apps & the Kosmos network
Casey Muratori's "Handmade Hero" series
Explorable Explanations
- Like the Parable of Polygons, a simulation about how sociology
- Cameras & Lenses, simulation about how cameras, light, and vision work
Internet art experiments like those done by Nolan (https://eieio.games/) & Neal (https://neal.fun/). See Nolan's newsletter
- Example of a simple research prototype that is only a few lines of code & a single HTML file, good & evil words

Overview of the series

(this article) How to run a piece of code in a plain HTML file on your computer, and deploy it on the internet with CodePen
Deploying a free static website or app on GitHub pages, like Ithaca Social Circle, or an Obsidian based website "digital garden"
- https://dev.to/omar4ur/handmade-github-pages-20pi
- mdBook template https://github.com/OmarShehata/mdbook-template/tree/main?tab=readme-ov-file#mdbook-template
Very basic backends ((1) using Google Sheets as a free backend, (2) or "BYO server" with remoteStorage, or (3) free CloudFlare object storage)
Basic user management with OAuth (twitter/google login)
Basic computer graphics visualizations (how to render pixels "raw", how to use a library like PixiJS or ThreeJS)
Basic data analysis (Jupyter notebooks & Google Colab)
- Simple twitter data NLP analysis
- ML cellular automata
Tips for navigating open source codebases, how to fork, edit, and contribute a pull request

In this lesson, you're going to setup this cellular automata simulation as a single HTML file on your computer, and copy/paste it onto CodePen to share it on the internet.

1. Install VS Studio Code

https://code.visualstudio.com/

We will use this as our IDE to edit code. The web browser is what will "run" the code.

You can use VS Code to edit any coding language, by either running the code outside of it, or finding a plugin that runs it for you.

2. Create an HTML file

This is just a text file with the extension .html. Inside of the HTML code we will embed the JavaScript code we want to run.

Create the file vanilla.html with VS Code, anywhere on your computer
Paste in the following code

<!DOCTYPE html>
<html>
<head>
    <meta charset="UTF-8" />
    <title>Vanilla JS</title>
</head>
<body>
    <p>Hello!</p>
    <script>
        // This is how you embed JavaScript inside of HTML
        console.log("Hello world!")
    </script>
</body>
</html>

Note: This series will not cover the basics of JS syntax. It's assumed you have a supplementary source or already know it. I recommend the book Eloquent Javascript, and the Codecademy JavaScript interactive lessons

Run this code by opening this file in your browser
Check the browser console to see if the JS code snippet ran correctly

Challenge: Can you make the web page display the current time?

You want to (1) get the current time in the JS code (2) Use document.querySelector() to select the HTML element and edit it with the local time (3) use setInterval to auto-update the code

Solution

Note: You can use the browser console as a REPL! You can expose a variable from your code by saying window.myVar = myVar, and then you have access to editing it live in your app

3. Deploy your code on the internet

This is the fastest way to get your code to run on millions of devices. It's "infinitely" scalable because the code runs on the user's computer, not on a central server. It's very cheap for CodePen, or any static host, because it's just delivering the code (consuming bandwidth, not CPU).

Make an account on https://codepen.io/
Create a new "pen"
Copy/paste your code into the "HTML" section of the CodePen interface
Click "Save" in the top right
To share it, click on the little icon of an arrow going out of a box, in the bottom right (on the same row as "Share" / "Export" / "Embed")

The shared link should look like this: https://codepen.io/omarshe7ta/full/wBWBqZq

You can continue writing code in the CodePen interface, or locally in VS Code.

4. Animated rectangle

Now we're going to run a piece of code that draws a rectangle that moves back and forth. Technically, the rectangle doesn't "move", we just erase the screen and redraw the rectangle at a new location at every frame.

Replace the code you have with this new template:

<!DOCTYPE html>
<html>
<head>
    <meta charset="UTF-8" />
    <title>Vanilla JS</title>
    <style>
        html, body {
            margin: 0px; overflow: hidden;
        }
    </style>
</head>
<body>
    <!-- the "canvas" element is the box on the page, inside of which all the drawing will happen -->
    <canvas></canvas>
    <script>
        // set the height & width to be full screen
        const canvas = document.querySelector("canvas");
        canvas.width = window.innerWidth;
        canvas.height = window.innerHeight;

        const ctx = canvas.getContext("2d");
        let positionX = 25
        function loop() {
            // Clear the screen
            ctx.clearRect(0, 0, canvas.width, canvas.height)
            // Draw a rectangle, x, y, width, height
            ctx.fillRect(positionX, 25, 20, 20);

            // move the position for the next frame
            positionX += 1

            // run the function "loop" again
            requestAnimationFrame(loop)
        }

        // run the function "loop" once
        loop()

    </script>
</body>
</html>

Challenge: Can you move the box faster?

Challenge: Can you make the box start at the right edge of the screen and move to the left

Challenge: Can you make the box move back & forth?

To do this, set the X of the box to be the sine of your count variable. Sine of any number will give you a range of [-1, 1], and you can scale that to the range you want it to go back and forth in.

Solution

5. Cellular Automata Grid

Here we're going to start with a piece of code that sets up a grid of pixels with a random color. We'll use this as a base to explore various "cellular automata" simulations, like the Game of Life.

Replace the code you have with this:

<!DOCTYPE html>
<html>
<head>
    <meta charset="UTF-8" />
    <title>Vanilla JS</title>
    <style>
        html, body {
            margin: 0px; overflow: hidden;
        }

    </style>
</head>
<body>
    <canvas></canvas>
    <script>
        const canvas = document.querySelector("canvas");
        canvas.width = window.innerWidth;
        canvas.height = window.innerHeight;
        const ctx = canvas.getContext("2d");

        const grid = {}
        const gridSize = 5
        initGrid()

        function loop() {
            requestAnimationFrame(loop)

            updateGrid()
            drawGrid()
        }
        loop()

        function initGrid() {
            // Fill the grid data structure with initial random colors
            for (let x = 0; x < canvas.width; x+= gridSize) {
                for (let y = 0; y < canvas.height; y+= gridSize) {
                    const color = Math.random() * 360
                    grid[`${x}_${y}`] = color
                }
            }
        }

        function updateGrid() {
            for (let x = 0; x < canvas.width; x+= gridSize) {
                for (let y = 0; y < canvas.height; y+= gridSize) {
                    let currentColor = grid[`${x}_${y}`]
                    // modify the color of the current square
                    // given the color it currently has
                    grid[`${x}_${y}`] = currentColor
                }
            }
        }

        function drawGrid() {
            // draw whatever is currently stored in the grid variable
            for (let x = 0; x < canvas.width; x+= gridSize) {
                for (let y = 0; y < canvas.height; y+= gridSize) {
                    const color = grid[`${x}_${y}`]
                    ctx.fillStyle = `hsl(${color}, 70%, 60%)`;
                    ctx.fillRect(x, y, gridSize, gridSize);
                }
            }
        }
    </script>
</body>
</html>

Try changing the gridSize variable at the top.

Challenge: Expose the grid variable to the browser console, so you can edit it in "real-time". Ask your LLM for a one-liner to take this data structure and set it all to 0, or some other constant

Challenge: Edit the updateGrid function to make the color change every frame

Challenge: Edit the updateGrid function to set the color of the current pixel, equal to the pixel to its left.

You'll need to handle the boundary condition to avoid errors.

Next lessons:

Deploy with GitHub Pages

Open source semantic embedding, search & clustering in NodeJS

Omar — Wed, 25 Sep 2024 15:43:24 +0000

This tutorial walks you through how to do semantic embedding completely offline with open source models in NodeJS. No knowledge of AI/ML is required. Source code: https://github.com/OmarShehata/minimal-embedding-template

An "embedding" is a high dimensional vector (x,y,z,...) that represents a concept. Think of it as the internal representation of words in an LLM. You can compute distances between these vectors.

Example: "Man" is much closer to "boy" & "woman", compared to "chicken". "Coffee" and "wifi" are somewhat close, and are both close to "coffee shop".

illustration of what these vectors might look like projected down to 2D

I think this is an extremely underutilized feature of modern LLM's, and it's much cheaper compute wise compared to text generation. Most of the time I don't really want the LLM to generate text as much as I want to see & manipulate the semantic concepts like this.

Setup

Clone the repo: https://github.com/OmarShehata/semantic-embedding-template. This contains a minimal NodeJS template that you can copy/paste and build on.

It uses (1) gpt4all as the LLM engine. This is where the open source model comes from, and is what converts a word/string/document into a vector. (2) Vectra as a local, single file vector database. Allows us to index & search vectors.

Run pnpm install, then run the first example in example-simple-embedding/index.js:

pnpm simple-embedding

This takes an array of strings and converts them to vectors:

await embeddings.insertText(['coffee shop', 'wifi', ...])

You can print the vectors with embeddings.getTextMap(). You can do a search as shown below. This returns a sorted list of the closest vectors in the DB, along with the cosine distance.

const results = await embeddings.search('coffee')
// returns:
// [
//   [ 'coffee shop', 0.8214959697396015 ],
//   [ 'wifi', 0.711907901740376 ],
//   [ 'hard work', 0.6709908415581982 ],
//   [ 'love peace & joy, relaxation', 0.6495931802131457 ]
// ]

(1 means it's exactly the same vector, -1 means it's exactly opposite, 0 means no correlation)

`embeddings` is a thin wrapper around gpt4all

lib/embeddings.js implements insertText which:

checks to see if these words are already in the DB
inserts them if they are not, with a batch update

The search function takes the query string and converts it to a vector, then runs a query with the vectra DB.

The specific model I'm using here is nomic-embed-text-v1.5 which is an open source model & free to use model that runs locally on your machine.

OpenAI embeddings

lib/embeddings-openai.js is a version of this file that has exactly the same API but sends the text to OpenAI. See OpenAI's embedding docs.

The OpenAI model captures more nuance in my experience (for example, it captures the semantic meaning of emojis whereas the open source one doesn't seem to).

Set the OPEN_API_KEY environment variable to use this. To run the example in example-openai-embedding/index.js:

pnpm openai-embedding

Clustering

To run example-clustering/index.js:

pnpm clustering

This clusters the vectors in the DB using k-means. You tell it the number of clusters you want to create, and it iterates over each point to find the "k nearest neighbors" to create these clusters.

Normally, you don't know how many clusters are in the data. There's various techniques to find this. One way is the "elbow method" where you cluster the dataset for increasingly higher cluster sizes and compute a "score". The score represents how close all items in the cluster are to a centroid. So the lower the score the more you end up with clusters of semantically related things.

I hope you found this useful! You can use the base code here to basically recreate Neal's Infinite Craft game. Basically put all the words in the dictionary in the vector database. Then to combine two words, add the vectors (or get the average?), then search for the closest thing to that combined vector.

This is my personal sandbox that I hope to add more stuff to. For example, there are models that can convert an image to a text description. You can then get a vector embedding for that text, and with that you can build an app where you can "CTRL+F" for your images (again, all offline, and free!)

Build & deploy your first web app

Omar — Fri, 14 Jul 2023 00:28:09 +0000

This guide is written for students going through CodeDay Labs. You will learn:

How to set up a simple web app from scratch
How to deploy your applications so others can access it via a URL

This tutorial is meant to teach the basic fundamentals of working with web technologies, regardless of what specific framework you might be using in the future.

Step 1 - Create an HTML page

We are going to create our first web page, by hand. You typically will not do this in your web development career, but it will help you understand what is happening under the hood, and what problems all these frameworks like React etc are trying to help you solve.

Create a new directory for our project. Name it web-dev-101
Open up your favorite code editor, like VSCode or Sublime Text, and create a new file called index.html
Copy/paste the following code:

<!DOCTYPE html>
<html>
<head>
    <meta charset="utf-8">
    <meta name="viewport" content="width=device-width, initial-scale=1">
    <title>Web Dev 101</title>
</head>
<body>
    <h1>Hello World!</h1>
</body>
</html>

Now preview your web page by double clicking on index.html, or dragging and dropping it into your web browser.

It should look like this:

Extra challenges:

What does the <h1> tag mean? What happens if you do <h2> tag instead? Or <p> tag?
Can you change the title in the browser tab, to make it say Web Dev 102 instead of Web Dev 101 ?

Step 2 - Add some styling / CSS

HTML is a markup language, it's for controlling what is shown on screen (as in, the content, the text, etc)

CSS is the styling language, it controls how things are displayed on screen (the color, the size, etc)

To add CSS/styling rules to your web page:

Add the following snippet inside the <head> tag:

    <style type="text/css">
        h1 {
            color: red;
        }
    </style>

It should go like this:

Then refresh your web page, the text should now be red:

This CSS rule we added makes ALL <h1> tags red. If we wanted to make a specific HTML element red, we have to give it either an ID or a class.

Give your <h1> element an id:

<h1 id="title">Hello World!</h1>

Change your CSS rule to target the element with the id title instead of all h1 tags:

#title {
  color: red;
}

The # symbol prefixing "title" is what tells CSS to apply the rule to any element that has the ID "title". This is called a CSS selector.

Another common type of CSS selector is a class. It works very similarly to an ID. You can give an element a class name as follows:

<h2 class="subtitle">My subtitle</h2>

And you can style it like this in CSS:

.subtitle {
  color: blue;
}

When should you use id vs class? Typically you use id for one specific element (like the title) and a class for something for which there can be multiple elements (like a paragraph, or a button).

Extra challenges:

What happens if you have an element that is BOTH an <h1> and has an id title, and you create a css rule that makes <h1> red and a css rule that makes title elements blue? Will the element be red or blue?
Can you use CSS to make an element underlined? Or bigger font?

Step 3 - Move your CSS to another file

You can add CSS into the same file as your HTML. But for bigger projects it is helpful to split it up into another file.

To do this:

Add the following line inside the <head> tag, this tells the HTML page it should load styles from a file called style.css:

<link rel="stylesheet" href="style.css">

Create a file called style.css and move your CSS in there. It might look like this:

Refresh the web page, confirm the CSS still works. Make a change to the style/color and confirm it updates when you refresh.

Step 4 - Deploy your web page

At this point the web page only exists on your computer. In order for others to access it on the internet, we need to:

Put it on a web server
Give it on a domain name

A "web server" is just a computer. Your personal laptop could be a web server. You just need to run a program that responds to HTTP requests and returns your web page.

In fact, let's try that real quick:

Install a simple http server library by running the following command:

npm install --global http-server

(if you don't already have npm installed, get it from here: https://nodejs.org/en/download. Yarn is also fine, the command might be slightly different.)

Open up a terminal window in the folder of your index.html and run the following command to run a web server on your laptop:

http-server .

The dot (.) tells it to serve the HTML files in the current directory.

This should give you an output that looks like this:

Available on:
  http://172.25.240.1:8080
  http://192.168.1.147:8080
  http://127.0.0.1:8080

Open up the second URL in your web browser (for me that's http://192.168.1.147:8080). You should see your website.
If it works, open up this URL on your phone or another computer. As long as it's connected to the same WiFi as your laptop, it should work.

What you just did is access your web page on your phone, powered by the web server running on your laptop!

You don't have a domain name, which is why you have to access it trough the IP address of your computer. It has to be on the same WiFi network for reasons outside the scope of this tutorial.

Step 5 - Deploy to a web server with a domain name

To put your web page on a server that others can access anywhere on the internet, we typically need to pay for this service. There's a few free options. One free option is: https://glitch.com/.

Create an account on https://glitch.com/signup
Create a new project (top right of the page)

Select the first option glitch-hello-website

This will take you to the Glitch code editor.

Copy paste your index.html from your computer into the Glitch editor (select index.html from the sidebar then just replace all its contents)
Do the same for style.css
Click the "Preview" button in the bottom panel to open up your website:

This is your website deployed to a web server, with a domain name you can now share!

Extra challenges:

Another option instead of using Glitch is using Github Pages. This is a way of automatically getting a deployed web page for any repository in your GitHub account.

The following guide will explain how to do it. Try to create a new repository for your web-dev-101 project and deploy it with GitHub pages:

https://pages.github.com/

Vite Hot Module Replacement - A Complete Example

Omar — Sun, 17 Apr 2022 15:45:21 +0000

This article explains how to set up hot module replacement (HMR) with Vite for a vanilla JS project. The goal is to be able to see changes in the code without refreshing or restarting the application.

For me this is particularly useful in creative coding when I'm iteratively building a simulation and don't want to lose state as I make live changes.

Complete source code for this article here: https://github.com/OmarShehata/vite-hot-reload-example.

Overview of my implementation

I'm going to show you how I've set up my HMR to make it as simple as possible to enable for new modules. In subsequent sections I'll break down how it works so you are aware of all the caveats and can use it in your project as needed.

To enable HMR on any module, I add the event handler at the top of the file:

import { HMREventHandler } from './HotModuleReloadSetup.js';

if (import.meta.hot) {
  import.meta.hot.accept(HMREventHandler)
}

This tells Vite to avoid refreshing the page when this module is updated, and instead fire an HMR event.

Next, I have a singleton instance of HotModuleReloadSetup that takes dynamically imported modules, and automatically swaps them out:

import HotModuleReloadSetup from './HotModuleReloadSetup.js';
// Setup HMR
const hmr = new HotModuleReloadSetup();
// Load a module that will be updated dynamically
hmr.import(await import('./Draw.js'));

I now have access to an instance of the Draw class through hmr.instances['Draw'].

Finally, the draw class defines a hotReload handler. This passes a reference to the old instance so you can copy over any state variables to the new instance.

So when I call hmr.instances['Draw'].draw() in the render loop, it will always use the latest code.

This pattern is modelled after how PlayCanvas does their hot reloading.

How it works

Here is what happens when you make a change in an HMR module (Draw.js in my case) & save the file:

1 - Vite triggers an HMR event (which we've added a listener to via import.meta.hot.accept)
2 - I then dispatch a custom event on the DOM with that new module (this is in HotModuleReloadSetup.js):

export function HMREventHandler(newModule) {
    const event = new CustomEvent('hot-module-reload', { detail: { newModule } });
    document.body.dispatchEvent(event);
}

This allows HotModuleReloadSetup to listen for these events globally.

3 - We search our map of modules to see if the new module we received exists
4 - If so, we create a new instance from the new module, call newInstance.hotReload(oldInstance), and discard the old module & old instance

swapModule(newModule) {
  const name = newModule.default.name;
  const oldModule = this.modules[name];
  const oldInstance = this.instances[name]
  if (!oldModule) return;

  const newInstance = new newModule.default();
  newInstance.hotReload(oldInstance)

  this.modules[name] = newModule
  this.instances[name] = newInstance
}

This way you can access either one instance of a class that is always updated through the instances map, or you can access the latest module through the modules map. The latter can be used if your code is spawning many instances of a class (like enemies or bullets), and you want your live code changes to affect newly spawned instances.

Caveats

If you need to have multiple instances of a class that are all updated when the code changes, you'll need to keep an array of these instances and swap them all out in the same way HotModuleReloadSetup.swapModule does
If your HMR class needs constructor arguments, you'll need to store those in the global manager that creates the new instances so they can be passed to them as well.
I turn off Rollup minification because it rewrites function names, but cannot update all uses of the function when using dynamic imports like this. That causes hmr.instances['Draw'].draw() to break because the function name is no longer draw in a production build.
I import the module by name and pass it to hmr.import() due to the limitations on dynamic imports in Vite, see: https://github.com/rollup/plugins/tree/master/packages/dynamic-import-vars#limitations

Thanks for reading! I hope you found this useful. If you find a bug or a better way to streamline HMR, especially for creative coding, I'd love to hear. You can find me on Twitter: https://twitter.com/omar4ur.

How to create bitmap fonts for Phaser JS with BMFont

Omar — Mon, 21 Feb 2022 12:55:21 +0000

This guide explains how to generate bitmap fonts from TTF or OTF files for use in PhaserJS. I'll be using BMFont which is Windows only.

Why bitmap fonts?

The main use case is if you're creating a pixel art game & want your text to match the retro style and have no antialiasing.

Below is an example from a recent game I made. The top is the standard font rendering in Phaser. The bottom is a bitmap version of the same font.

Note the antialiasing inside letters like g, a, or o in the top image. This creates what looks like blurry artifacts at the small scale of pixel art games. The bottom image has the crisp, pixelated rendering expected in retro games.

Bitmap fonts may also be faster to render. From the Phaser docs:

BitmapText objects are less flexible than Text objects, in that they have less features such as shadows, fills and the ability to use Web Fonts, however you trade this flexibility for rendering speed

To use a bitmap font, Phaser needs:

An image containing all the possible characters
An XML file that defines the x/y/width/height of each character in the image.

It's basically a spritesheet of letters.

Here's an example I generated from this public domain font: https://www.fontspace.com/public-pixel-font-f72305.

The image with all the letters:

Snippet from the XML:

<char id="32" x="507" y="0" width="3" height="1" xoffset="-1" yoffset="31" xadvance="32" page="0" chnl="15" />
<char id="33" x="285" y="87" width="18" height="28" xoffset="3" yoffset="4" xadvance="32" page="0" chnl="15" />
<char id="34" x="441" y="108" width="22" height="16" xoffset="3" yoffset="4" xadvance="32" page="0" chnl="15" />

You can download this generated example in the format that Phaser can use here: pixel_bitmap_font.zip

Step 1 - Download BMFont

Download the executable on this page:

https://www.angelcode.com/products/bmfont/

Step 2 - Load the font

Prepare your font as a TTF file or similar
Open up bmfont64.exe
Select Options > Font settings
Select your font file in Add font file
Then select the name of the font in the Font dropdown

If your font is installed system-wide you can skip the Add font file step and just select the name of the font directly.

Now you should see your font loaded:

Step 3 - Export

First, we change the export settings:

Select Options > Export options
Select XML as the Font descriptor
Select PNG as the textures option

Press OK

Then to export:

Select all letters for export (Ctrl + A)
File > Save bitmap font as

This will generate an XML file (it'll have the extension .fnt, you can rename that to .xml or leave it as-is, Phaser will be able to read it as an XML either way) and a PNG file.

You may need to increase the width/height in the export options to keep all the letters in one image.

Step 4 - Use it in Phaser

Tell Phaser where to load the PNG & XML files:

// Load it
this.load.bitmapFont('bitmapFontName', 'font.png', 'font.fnt');
// Add it to the scene
this.add.bitmapText(0, 0, 'bitmapFontName', 'Lorem ipsum\ndolor sit amet');

Full example here: https://labs.phaser.io/edit.html?src=src/loader/bitmap%20text/load%20bitmap%20text.js.

Final thoughts

Note that a generated bitmap font has a font size baked in. Phaser can scale the font up and down but that may introduce artifacts in some cases. If you know the font size you want ahead of time you can set it in Options > Font settings.

I used a font size of 32px in my game which was big enough so that it still looked good when scaled down or up a bit.

I hope you found this useful! If you have any corrections or find a better way to generate bitmap fonts for Phaser I'm happy to update this article. Find me on Twitter (@Omar4ur) or my website (https://omarshehata.me/).

Create a satellite tracker from scratch in 30 lines of JavaScript

Omar — Thu, 06 May 2021 02:32:25 +0000

This tutorial will walk you through how to create a web app that visualizes the location of any satellite in real-time, like the International Space Station.

We're going to do this from scratch, using the same techniques a real rocket scientist would!

We'll look at where to get the data about individual satellites that the government publishes, known as Two-Line Element Sets, or TLE's.
We'll use satellite-js to predict the orbit of the satellite given the TLE's (this is the rocket science part).
We'll use CesiumJS to visualize the result, but you can use any library/engine that can take in longitude, latitude, and height.

Below is a preview of the final result.

This is showing the path of the International Space Station, sped up by 40x. To see its current location in real-time, click the clock icon at the top-left of the clock wheel.

Here's a direct link to the app. And the source code on Glitch.

1 - Get the satellite's Two-Line Element Set

Given this description of the orbit, we can predict the location of where it's going to be at any moment in time (which is step 2 below).

This means that most "live" satellite trackers are not live in the same way that tracking a delivery car on a map is. Instead of relying on constantly receiving position updates, those who track objects in space will often get the latest TLE's (which are regularly updated) and use that to predict where the object is right now.

Where do we get the TLE's? There is no one global official registry. Whoever owns the satellite and is monitoring it is responsible for updating and publishing the TLE for the benefit of the global space community (unless it's a spy satellite).

We can find these TLE's on Space Track which is a registry run by the United States Space Command.

Another source is this list on CeleStrak maintained by Dr. T.S. Kelso.

We're going to use CeleStrak since it doesn't require a login. To find the TLE for the International Space Station, click on the Space Stations link.

The first one is the TLE for the ISS:

ISS (ZARYA)             
1 25544U 98067A   21122.75616700  .00027980  00000-0  51432-3 0  9994
2 25544  51.6442 207.4449 0002769 310.1189 193.6568 15.48993527281553

The meaning of these numbers is listed in table 1 in Dr T.S. Kelso's column. Most of them are identifiers and metadata about the satellite, like when it was launched.

You can find TLE's for weather satellites, GPS satellites, and even SpaceX's Starlink constellation in this same format.

2 - Predict the satellite orbit

Now that you know how to get the TLE of the object you're interested in tracking, the next step is converting that to a position in time.

We're going to use satellite-js for this.

Include the library from a CDN:

<script src="https://cdnjs.cloudflare.com/ajax/libs/satellite.js/4.0.0/satellite.min.js"></script>

Then pass the TLE to it, and a time:

const ISS_TLE = 
    `1 25544U 98067A   21122.75616700  .00027980  00000-0  51432-3 0  9994
     2 25544  51.6442 207.4449 0002769 310.1189 193.6568 15.48993527281553`;
// Initialize the satellite record with this TLE
const satrec = satellite.twoline2satrec(
  ISS_TLE.split('\n')[0].trim(), 
  ISS_TLE.split('\n')[1].trim()
);
// Get the position of the satellite at the given date
const date = new Date();
const positionAndVelocity = satellite.propagate(satrec, date);
const gmst = satellite.gstime(date);
const position = satellite.eciToGeodetic(positionAndVelocity.position, gmst);

console.log(position.longitude);// in radians
console.log(position.latitude);// in radians
console.log(position.height);// in km

Now we have the position of the satellite at the current time, new Date().

This position is produced as a result of simulating a specific model of satellite motion. This model is called SGP4/SDP4. All TLE's assume this specific model.

If you're wondering about the accuracy of this model, the short answer is, it depends.

Accuracy of the two-line element sets is dependent upon a number of factors. These range from the particular sensors used and amount of data collected to the type of orbit and condition of the space environment. Unfortunately, since these factors vary for each element set, so does the accuracy. While NORAD has experimented with methods to incorporate prediction quality into the element sets, none of these methods has yet proved successful.

3 - Visualize the result

Now we have a way to get the location of any satellite, at any given time. We can pass in future times to animate its path, which we'll do in the next step.

First, let's see how to visualize an individual point in space using CesiumJS.

We load the library from CDN:

<script src="https://cesium.com/downloads/cesiumjs/releases/1.81/Build/Cesium/Cesium.js"></script>
  <link href="https://cesium.com/downloads/cesiumjs/releases/1.81/Build/Cesium/Widgets/widgets.css" rel="stylesheet">

And create a container element:

<div id="cesiumContainer"></div>

We then initialize the viewer. Here we pass in some extra options to disable functionality that requires an access token:

// Initialize the Cesium viewer.
const viewer = new Cesium.Viewer('cesiumContainer', {
  imageryProvider: new Cesium.TileMapServiceImageryProvider({
    url: Cesium.buildModuleUrl("Assets/Textures/NaturalEarthII"),
  }),
  baseLayerPicker: false, geocoder: false, homeButton: false, infoBox: false,
  navigationHelpButton: false, sceneModePicker: false
});
viewer.scene.globe.enableLighting = true;

Finally, we'll visualize the satellite position as a red dot in space:

const satellitePoint = viewer.entities.add({
  position: Cesium.Cartesian3.fromRadians(
    position.longitude, position.latitude, position.height * 1000
  ),
  point: { pixelSize: 5, color: Cesium.Color.RED }
});

See the complete source code of this step in simple.html on Glitch.

4 - Animate the path

To animate the path, we just need to sample more positions in the future. CesiumJS has a built-in way to interpolate between these samples over time.

The setup for this is a bit verbose. You can see the full code on Glitch. The important concepts are described below.

We create a SampledPositionProperty. This is an object that will hold position samples over time and will interpolate between them:

const positionsOverTime = new Cesium.SampledPositionProperty();

We loop through however many samples we want to get, and for each sample, we construct a time object, called JulianDate in CesiumJS, and a position, and we add that as a sample:

for (let i = 0; i < totalSeconds; i+= timestepInSeconds) {
  const time = Cesium.JulianDate.addSeconds(start, i, new Cesium.JulianDate());
  // ...Get position from satellite-js...
  const position = Cesium.Cartesian3.fromRadians(p.longitude, p.latitude, p.height * 1000);
  positionsOverTime.addSample(time, position);
}

Finally, we pass this positionsOverTime object to our point.

const satellitePoint = viewer.entities.add({
  position: positionsOverTime,
  point: { pixelSize: 5, color: Cesium.Color.RED }
});

The point will move as the timeline at the bottom moves. To attach the camera to the moving point we do:

viewer.trackedEntity = satellitePoint;

Conclusion

I hope you enjoyed learning a little bit about what goes into building a satellite tracker. There's a lot more to the topic we didn't touch on, like what exactly do the parameters in the TLE mean? How often are they updated? How are they updated?

I don't know, but I find it really empowering to know what formats this kind of data is published in & where to get it, and quite amazing that we can do all this directly in the browser with JavaScript!

Here's a couple fun ideas to explore now that we can do this:

Visualize multiple satellites, like the entire Starlink constellation. Inspired by Celestrak's viewer which shows every satellite in its catalog. Perhaps visualize how the number of Starlink satellites grew over time?

Or simulate what it would look from street level. Maybe add buildings/elevation data to find the best place in the city to see the satellite?

There's a prototype of this in street-level.html in the Glitch source code. Demo: https://satellite-viewer.glitch.me/street-level.html.

See also James Darpinian's "See a satellite tonight" app which uses a combination of CesiumJS and Google street view.

It might also be fun to use a 3D model of the right scale instead of a dot, and get a real sense of how close satellites get to each other in space.

How to load Sketchfab models directly in a ThreeJS app

Omar — Wed, 28 Apr 2021 01:34:15 +0000

Sketchfab's API gives you programmatic access to the largest collection of glTF 3D models on the web. This article walks you through a minimal code example to show you how to:

Let your users authenticate with the Sketchfab API
Download a 3D model as a zip file containing the glTF
Load this zip file into ThreeJS

Source code: https://github.com/OmarShehata/threejs-sketchfab-example

How it works

I originally implemented this to let readers of my WebGL outlines tutorial see how the effect looked on test cases of their choosing. Since I kept finding algorithms that didn't work on my specific corner cases (but I wouldn't find out until after I implemented it/downloaded & ran it!)

It's a really easy way to let users bring in their own data (or millions of example models).

You can see how this works in this outlines live demo:

Open the demo: https://omarshehata.github.io/csb-l01dp/
Click Login to Sketchfab
When directed back to the app, paste a link of a model in the Sketchfab URL field, like this: https://sketchfab.com/3d-models/skull-downloadable-1a9db900738d44298b0bc59f68123393

The model must be downloadable by the Sketchfab account that's logged in.

1 - Authenticate with the Sketchfab API

The first step is to register your app with Sketchfab. The instructions for this are here:

https://sketchfab.com/developers/oauth#registering-your-app

The current process at the time of writing is to contact them.

You'll need to pick a redirect URI. This should be the final URI where you will deploy your app.

You'll want to use the Implicit grant type. We can't keep an API key a secret in a web app, so we instead rely on the redirect URI (if someone malicious uses your client ID, Sketchfab will redirect them to your real app after login, regardless of who may have initiated the login).

After your register your app you'll have a Client ID.
You'll use this to send the user to Sketchfab for login as shown here:

const CLIENT_ID = 'YOUR_CLIENT_ID_HERE';
const AUTHENTICATION_URL = `https://sketchfab.com/oauth2/authorize/?state=123456789&response_type=token&client_id=${CLIENT_ID}`;
window.open(AUTHENTICATION_URL, '_blank');

Once the login is done, you'll need to grab the access token from the URL. Here's a snippet that does this and stores it in local storage here:

checkToken() {
    // Check if there's a new token from the URL
    const url = new URL(window.location)
    // Extract the token and save it
    const hashParams = url.hash.split('&');
    for (let param of hashParams) {
        if (param.indexOf("access_token") !== -1) {
            const token = param.replace('#access_token=', '');
            console.log("Detected Sketchfab token: ", token);
            localStorage.setItem("sb_token", token);
        }
    }

    // Load token from local storage
    this.token = localStorage.getItem("sb_token");
}

Source code on GitHub.

You'll use this token for subsequent API calls.

Note: while you're developing locally, Sketchfab will redirect to the production URI. You'll need to copy the URI params to your localhost to test.

2 - Download the 3D model

Once you have a token, you can use this to fetch a download URI for the glTF model.

Here is the code snippet for fetching the download URI given any Sketchfab URI like this: https://sketchfab.com/3d-models/skull-downloadable-1a9db900738d44298b0bc59f68123393

async getModelDownloadUrl(inputUrl) {
    // Extract the model ID from the URL
    const input = new URL(inputUrl);
    // The ID is always the last string when seperating by '-'
    const pieces = input.pathname.split('-');
    const modelID = pieces[pieces.length - 1];

    const metadataUrl = `https://api.sketchfab.com/v3/models/${modelID}/download`;
    const options = {
        method: 'GET',
        headers: {
            Authorization: `Bearer ${this.token}`,
        },
        mode: 'cors'
    };

    // This call will fail if model can't be downloaded
    const response = await fetch(metadataUrl, options);
    const metadata = await response.json();
    return metadata.gltf.url;
}

Source code on GitHub.

You can filter for downloadable models in the Sketchfab search:

This will show you either free models you can download or models that can be purchased.

Now that we have a URL to a zip file containing a glTF model, we can pass that to ThreeJS to load it.

3 - Load the ZIP file into ThreeJS

This was the tricky part of me. Normally ThreeJS requires a direct URL to the glTF file. To load it from a zip file we're going to (1) unzip the contents into memory using JSZip.

We can't just pass the unzipped raw content to ThreeJS, because a glTF file may reference other files by a filepath (image or geometry resources). So we need to (2) create a Blob for each resource and (3) override the resources the glTF file is requesting with the Blob URI's.

For example, if the glTF file is trying to load textures/defaultMat_baseColor.jpeg as a relative filepath, we detect this, and instead pass this URI:

//Unzip from JSZip
const file = unzippedBaseColorJPEGFile;
// Create a Blob from this file
const blob = await file.async('blob');
// Create a URL to this Blob
const baseColorBlobUrl = URL.createObjectURL(blob);
// Use ThreeJS's loading manager so instead of loading from relative filepaths we load from the blob URI's we created
const loadingManager = new THREE.LoadingManager();                            loadingManager.setURLModifier((url) => {
  if (url == 'textures/defaultMat_baseColor.jpeg') {
    return baseColorBlobUrl;
  }
});

We use ThreeJS's LoadingManager to do this.

Here is the code snippet that can take a URL to any zipped glTF and load it in ThreeJS:

async readZip(zipUrl, scene) {
    const response = await fetch(zipUrl);
    checkStatus(response);
    const arrayBuffer = await response.arrayBuffer();

    const result = await JSZip.loadAsync(arrayBuffer);

    const files = Object.values(result.files).filter(item => !item.dir);
    const entryFile = files.find(f => getExtension(f.name) === 'gltf');
    // Create blobs for every file resource
    const blobUrls = {};
    for (const file of files) {
        console.log(`Loading ${file.name}...`);
        blobUrls[file.name] = await getFileUrl(file);
    }
    const fileUrl = blobUrls[entryFile.name];

    scene.clear();
    // Re-add the light
    const light = new THREE.DirectionalLight(0xffffff, 1);
    scene.add(light);
    light.position.set(1.7, 1, -1);

    const loadingManager = new THREE.LoadingManager();
    loadingManager.setURLModifier((url) => {
        const parsedUrl = new URL(url);
        const origin = parsedUrl.origin;
        const path = parsedUrl.pathname;
        const relativeUrl = path.replace(origin + '/', "");

        if (blobUrls[relativeUrl] != undefined) {
            return blobUrls[relativeUrl];
        }

        return url
    });
    const gltfLoader = new GLTFLoader(loadingManager);
    gltfLoader.load(fileUrl, (gltf) => {
      scene.add(gltf.scene);
    });
}

Source code on GitHub.

4 - Display attribution

Using 3D models from Sketchfab in your application requires that you display attribution to the original author. Read more about this on Sketchfab.

We can automatically get the attribution & license information from this Data API route.

Here is a function that will construct the attribution text given a modelID:

async getAttributionText(modelID) {
        const modelDataUrl = `https://api.sketchfab.com/v3/models/${modelID}`;
        const options = {
            method: 'GET',
            headers: {
                Authorization: `Bearer ${this.token}`,
            },
            mode: 'cors'
        };
        const response = await fetch(modelDataUrl, options);
        const metadata = await response.json();

        const license = { name: metadata.license.label, url: metadata.license.url };
        const user = { name: metadata.user.displayName , url: metadata.user.profileUrl };
        const model = { name: metadata.name, url: metadata.viewerUrl };
        const attributionText = 
        `This work is based on <a href="${model.url}" target=_blank>${model.name}</a>
        by <a href="${user.url}" target=_blank>${user.name}</a> 
        licensed under <a href="${license.url}" target=_blank>${license.name}</a>.`;

        return attributionText;
    }

Source code on GitHub.

The example app will display the attribution in the bottom left corner, linking to the original model URL, author's Sketchfab profile, and the license.

Known issues

Normalizing and scaling models when loading them in ThreeJS would help solve this, similar to how Don McCurdy's glTF Viewer works.

Thanks for reading! If you found this helpful, follow me on Twitter @omar4ur to see more of my work. Other ways to reach me at https://omarshehata.me/.

How to hire a lawyer in the court of public opinion

omar — Mon, 30 Mar 2026 16:24:07 GMT

Last Monday I announced that I have been running a company, now with a bit of formal paperwork, and it’s called “Prosocial Engineering”.

Starting a company - devlog

omar — Mon, 23 Mar 2026 16:19:52 GMT

Good morning!

How we grade presentation night

omar — Thu, 18 Dec 2025 16:01:30 GMT

Presentation Night is a ~monthly event I host here in Ithaca.

Beginners can do things that experts cannot

omar — Wed, 16 Jul 2025 14:50:44 GMT

10 years ago I wrote “A Beginner's Guide to Coding Graphics Shaders”. It remained the top google search result for ~7 years for the generic term “graphics shader tutorial”. It had hundreds of comments saying:

Hulu is A/B testing ads in the pause screen — I hate it

omar — Sun, 29 Sep 2024 22:04:47 GMT

500 words, 2 minute read

Washington Post is collecting TikTok user data from its followers for public good

omar — Fri, 20 Sep 2024 13:21:41 GMT

I wrote about communities collecting twitter user data so that we can collectively learn about ourselves, and now the Washington Post is doing the same thing!!

Twitter shut off API access; users are now volunteering their data for an open API

omar — Wed, 18 Sep 2024 12:19:34 GMT

A lot of community applications died when API access to twitter, reddit & others was severely restricted, but they can't stop us from sharing our own data with each other!

What exactly is ChatGPT, if it's not a chatbot?

omar — Tue, 27 Aug 2024 23:24:57 GMT

My rewrite of Scott Alexander's "Janus' Simulators" about the nature of LLM's

Fighting twitter's censorship of substack

omar — Sat, 06 Jul 2024 20:48:12 GMT

Twitter refuses to display the nice preview card for substack links, but this a silly ban that is easily defeated with some basic understanding of how the internet works

Making software by "cosmic pizza order"

omar — Fri, 28 Jun 2024 18:13:09 GMT

I used to bemoan the fact that every time I post about an idea someone tells me, "that's been done before!" — now I've learned to love it

My favorite bench in Ithaca NY

omar — Tue, 11 Jun 2024 01:20:48 GMT

"to those who shall sit here..."

I love Scott Hanselman's social media presence

omar — Thu, 06 Jun 2024 14:54:30 GMT

Everyone likes to hate on social media for being toxic/polarizing, but, apparently it doesn't have to be this way

My favorite 1980's Canadian TV show: Bits and Bytes

omar — Sat, 11 May 2024 23:11:45 GMT

Reflections on whatever I'm thinking about that week (the web, culture, computer graphics, math etc)

Presentation Night - Feb 7 - Recap

omar — Sun, 02 Apr 2023 17:47:35 GMT

Highlights from the first Presentation Night