Process
Once the file(s) pass upload completion and post-upload validation they enter process state.
Processing steps are determined by filetype uploaded, but there is some overlap (e.g. photos extracted from videos enter the image processing pipeline).

pro-001u - EAC GoPro Video (.360)

.360 are a GoPro proprietary format (very similar to Google EAC). As such, we need a few more processing steps to first convert a "regular" mp4 equirectangular video

1. Extract metadata

A raw copy of the video telemetry is required to be injected into extracted frames at a later step. This is done using exiftool
1
exiftool -ee -G3 -api LargeFileSupport=1 -X VIDEO.360 > VIDEO_META.xml
Copied!

2. Convert to frames

Now the video can be converted into two sets of frame (at 0.5fps)
1
$ ffmpeg -i INPUT.360 -map 0:0 -r 0.5 -q:v 1 track0/img%d.jpg -map 0:5 -r 0.5 -q:v 1 track5/img%d.jpg
Copied!
Note: 2 images are created per frame. For more info about why this happens, read more about the GoPro 360 file format: https://www.trekview.org/blog/2021/reverse-engineering-gopro-360-file-format-part-1/

3. Save output to S3

4. Merge frames into equirectangular images

We use MAX2Sphere to do this: https://github.com/trek-view/MAX2sphere-batch
1
$ MAX2sphere-batch [options] track%d/frame%4d.jpg
Copied!
There are only two output width variations possible (only 2 settings for .360 on camera):
Use a GoPro Max 3K video (input width = 2272)
1
$ @SYSTEM_PATH/MAX2sphere -w 2272 track%d/GS018423_%d.jpg
Copied!
Use a GoPro Max 5.2K video (input width = 4096)
1
$ @SYSTEM_PATH/MAX2sphere -w 4096 track%d/GS018421_%d.jpg
Copied!

5. Inject metadata (inc telemetry) into each frame

At this point we inject the following metadata tags to the image:
Metadata field
Example extracted
GPS:GPSDateStamp
2020:04:13
GPS:GPSTimeStamp
15:37:22.444
GPS:GPSLatitude
51 deg 14' 54.51"
GPS:GPSLatitudeRef
North
GPS:GPSLongitude
16 deg 33' 55.60"
GPS:GPSLongitudeRef
West
GPS:GPSAltitudeRef
Above Sea Level
GPS:GPSAltitude
157.641 m
XMP-GPano:StitchingSoftware
Spherical Metadata Tool
XMP-GPano:SourcePhotosCount
2
XMP-GPano:UsePanoramaViewer
true
XMP-GPano:ProjectionType
equirectangular
XMP-GPano:CroppedAreaImageHeightPixels
(defined by height of output image at step 3)
XMP-GPano:CroppedAreaImageWidthPixels
(defined by width of output image at step 3)
XMP-GPano:FullPanoHeightPixels
(defined by height of output image at step 3)
XMP-GPano:FullPanoWidthPixels
(defined by width of output image at step 3)
XMP-GPano:CroppedAreaLeftPixels
0
XMP-GPano:CroppedAreaTopPixels
0
IFD0:Model
GoPro Max

6. Save final jpgs to S3

The S3 photo url returned on successful upload is stored against photo record and DB.

7. Enter image processing workflow

We can now process the video frames as we stitched equirectangular images: pro-007e - Stitched equirectangular images (jpg)

pro-002u - Stitched equirectangular video (.mp4)

1. Extract metadata

A raw copy of the video telemetry is required to be injected into extracted frames at a later step. This is done using exiftool
1
exiftool -ee -G3 -api LargeFileSupport=1 -X VIDEO.mp4 > VIDEO_META.xml
Copied!

2. Break into frames

1
ffmpeg -i VIDEO.mp4 -r 0.5 -q:v 1 img%d.jpg
Copied!

3. Save output to S3

4. Inject metadata (inc telemetry) into each frame

At this point we inject the following metadata tags to the image:
Metadata field
Example extracted
GPS:GPSDateStamp
2020:04:13
GPS:GPSTimeStamp
15:37:22.444
GPS:GPSLatitude
51 deg 14' 54.51"
GPS:GPSLatitudeRef
North
GPS:GPSLongitude
16 deg 33' 55.60"
GPS:GPSLongitudeRef
West
GPS:GPSAltitudeRef
Above Sea Level
GPS:GPSAltitude
157.641 m
XMP-GPano:StitchingSoftware
Spherical Metadata Tool
XMP-GPano:SourcePhotosCount
2
XMP-GPano:UsePanoramaViewer
true
XMP-GPano:ProjectionType
equirectangular
XMP-GPano:CroppedAreaImageHeightPixels
2688
XMP-GPano:CroppedAreaImageWidthPixels
5376
XMP-GPano:FullPanoHeightPixels
2688
XMP-GPano:FullPanoWidthPixels
5376
XMP-GPano:CroppedAreaLeftPixels
0
XMP-GPano:CroppedAreaTopPixels
0
IFD0:Model
GoPro Max

5. Save final jpgs to S3

The S3 photo url returned on successful upload is stored against photo record and DB.

6. Enter image processing workflow

We can now process the video frames as we stitched equirectangular images: pro-007e - Stitched equirectangular images (jpg)

pro-003u - 2 unstitched equirectangular videos (mp4's)

1. Extract metadata from front video

A raw copy of the video telemetry (contained in front video) is required to be injected into extracted frames at a later step. This is done using exiftool
1
exiftool -ee -G3 -api LargeFileSupport=1 -X GPFR0001.mp4 > GPFR0001.xml
Copied!

2. Convert both videos to frames

Now both videos (front and back) can be split into frames (FRont and BacK)
1
ffmpeg -i GPFR0001.mp4 -r 0.5 -q:v 1 FR/FRimg%d.jpg
2
ffmpeg -i GPBK0001.mp4 -r 0.5 -q:v 1 BK/BKimg%d.jpg
Copied!

3. Save output to S3

4. Merge frames into equirectangular images

We use fusion2sphere to do this: https://github.com/trek-view/fusion2sphere
1
fusion2sphere -w 4096 -b 5 FR/ BK/ parameter.txt
Copied!
There are only two output width variations possible (only 2 settings for unstitched mp4 video on camera):
Use a GoPro Fusion 5.2k video frame (each frame width = 2704)
1
@SYSTEM_PATH/fusion2sphere -b 5 -f testframes/5_2k/FR/GPFR7152_img1.jpg testframes/5_2k/BK/GPBK7152_img1.jpg -o testframes/18mp/GP7152.jpg parameter-examples/video-5_2k-mode.txt
Copied!
Use a GoPro Fusion 3k video frame (each frame width = 1568)
1
@SYSTEM_PATH/fusion2sphere -b 5 -f testframes/3k/FR/GPFR0003_img1.jpg testframes/3k/BK/GPBK0003_img1.jpg -o testframes/18mp/GP0003.jpg parameter-examples/video-3k-mode.txt
Copied!

5. Inject metadata (inc telemetry) into each frame

At this point we inject the following metadata tags to the image:
Metadata field
Example extracted
GPS:GPSDateStamp
2020:04:13
GPS:GPSTimeStamp
15:37:22.444
GPS:GPSLatitude
51 deg 14' 54.51"
GPS:GPSLatitudeRef
North
GPS:GPSLongitude
16 deg 33' 55.60"
GPS:GPSLongitudeRef
West
GPS:GPSAltitudeRef
Above Sea Level
GPS:GPSAltitude
157.641 m
XMP-GPano:StitchingSoftware
Spherical Metadata Tool
XMP-GPano:SourcePhotosCount
2
XMP-GPano:UsePanoramaViewer
true
XMP-GPano:ProjectionType
equirectangular
XMP-GPano:CroppedAreaImageHeightPixels
(defined by height of output image at step 3)
XMP-GPano:CroppedAreaImageWidthPixels
(defined by width of output image at step 3)
XMP-GPano:FullPanoHeightPixels
(defined by height of output image at step 3)
XMP-GPano:FullPanoWidthPixels
(defined by width of output image at step 3)
XMP-GPano:CroppedAreaLeftPixels
0
XMP-GPano:CroppedAreaTopPixels
0
IFD0:Model
GoPro Fusion

6. Save final jpgs to S3

The S3 photo url returned on successful upload is stored against photo record and DB.

7. Enter image processing workflow

We can now process the video frames as we stitched equirectangular images: pro-007e - Stitched equirectangular images (jpg)

pro-004u - Hero video (.mp4)

1. Extract metadata

A raw copy of the video telemetry is required to be injected into extracted frames at a later step. This is done using exiftool
1
exiftool -ee -G3 -api LargeFileSupport=1 -X VIDEO.mp4 > VIDEO_META.xml
Copied!

2. Break into frames

1
ffmpeg -i VIDEO.mp4 -r 0.5 -q:v 1 img%d.jpg
Copied!

3. Save output to S3

4. Inject metadata (inc telemetry) into each frame

At this point we inject the following metadata tags to the image:
Metadata field
Example extracted
GPS:GPSDateStamp
2020:04:13
GPS:GPSTimeStamp
15:37:22.444
GPS:GPSLatitude
51 deg 14' 54.51"
GPS:GPSLatitudeRef
North
GPS:GPSLongitude
16 deg 33' 55.60"
GPS:GPSLongitudeRef
West
GPS:GPSAltitudeRef
Above Sea Level
GPS:GPSAltitude
157.641 m
IFD0:Model
GoPro Max

5. Save final jpgs to S3

The S3 photo url returned on successful upload is stored against photo record and DB.

6. Enter image processing workflow

We can now process the video frames as we Hero photos: pro-006e - HERO images

pro-005u - Unstitched equirectangular images (jpg)

1. Extract metadata from all front photos

A raw copy of the GPS (contained in front video) is required to be injected into extracted frames at a later step. This is done using exiftool
1
exiftool -ee -X GF075169.JPG > GF075169.xml
Copied!
Note, this is done for every front photo in sequence

2. Merge frames into equirectangular images

We use dualfish2sphere to do this: https://github.com/trek-view/dualfish2sphere
1
dualfish2sphere [options] track0filename track5filename
Copied!
There is only one output width variations possible (only 1 settings for unstitched jpg photo on camera):
Use a GoPro Fusion 18mp photo (input frame width = 3104)
1
@SYSTEM_PATH/fusion2sphere -b 5 -f testframes/18mp/FR/GF075169.JPG testframes/18mp/BK/GB075169.JPG -o testframes/18mp/G075169.jpg parameter-examples/photo-mode.txt
Copied!

3. Inject metadata (inc telemetry) into each frame

At this point we inject the following metadata tags to the image:
Metadata field
Example extracted
GPS:GPSDateStamp
2020:04:13
GPS:GPSTimeStamp
15:37:22.444
GPS:GPSLatitude
51 deg 14' 54.51"
GPS:GPSLatitudeRef
North
GPS:GPSLongitude
16 deg 33' 55.60"
GPS:GPSLongitudeRef
West
GPS:GPSAltitudeRef
Above Sea Level
GPS:GPSAltitude
157.641 m
XMP-GPano:StitchingSoftware
Spherical Metadata Tool
XMP-GPano:SourcePhotosCount
2
XMP-GPano:UsePanoramaViewer
true
XMP-GPano:ProjectionType
equirectangular
XMP-GPano:CroppedAreaImageHeightPixels
(defined by height of output image at step 2)
XMP-GPano:CroppedAreaImageWidthPixels
(defined by width of output image at step 2)
XMP-GPano:FullPanoHeightPixels
(defined by height of output image at step 2)
XMP-GPano:FullPanoWidthPixels
(defined by width of output image at step 2)
XMP-GPano:CroppedAreaLeftPixels
0
XMP-GPano:CroppedAreaTopPixels
0
IFD0:Model
GoPro Fusion

4. Save final jpgs to S3

The S3 photo url returned on successful upload is stored against photo record and DB.

5. Enter image processing workflow

We can now process the video frames as we stitched equirectangular images: pro-007e - Stitched equirectangular images (jpg)

pro-006e - HERO (non-equirectangular) images

1. Add watermark

Users will upgrade account to remove Trek View watermark from their flat images. Therefore we create a seconds "branded copy" with the default Trek View logo (in addition to non-watermark copy at step 1).
Watermark width is 10% of image height and placed in bottom right corner.
The following is used:
trek-view-square-nadir.png
trek-view-square-nadir.png
39KB
Image
\

2. Write metadata to DB

For each image in a sequence, we keep a record of metadata in the database.
The following metadata is extracted and captured:
Image metadata field extracted
Example extracted
GPS:GPSDateStamp
2020:04:13
GPS:GPSTimeStamp
15:37:22.444
GPS:GPSLatitude
51 deg 14' 54.51"
GPS:GPSLatitudeRef
North
GPS:GPSLongitude
16 deg 33' 55.60"
GPS:GPSLongitudeRef
West
GPS:GPSAltitudeRef
Above Sea Level
GPS:GPSAltitude
157.641 m
IFD0:Model
GoPro Max

3. Proceed to Final steps

Now final calculations can be preformed. See final processing steps: pro-008e - Final processing steps

pro-007e - Stitched equirectangular images (jpg)

1. Add nadir

Users will upgrade account to remove Trek View nadir from their equirectangular images. Therefore we create a seconds "branded copy" with the default Trek View nadir (in addition to non-nadir copy at step 1).
Nadir size should be 15% of image height and placed at base of equirectangular image.
Processing images will retain all original image metadata.
The following file is used:
trek-view-square-nadir-6.png
trek-view-square-nadir-6.png
56KB
Image

2. Write metadata to DB

For each image in a sequence, we keep a record of metadata in the database.
The following metadata is extracted and captured:
equirectangular only?
Image metadata field extracted
Example extracted
FALSE
GPS:GPSDateStamp
2020:04:13
FALSE
GPS:GPSTimeStamp
15:37:22.444
FALSE
GPS:GPSLatitude
51 deg 14' 54.51"
FALSE
GPS:GPSLatitudeRef
North
FALSE
GPS:GPSLongitude
16 deg 33' 55.60"
FALSE
GPS:GPSLongitudeRef
West
FALSE
GPS:GPSAltitudeRef
Above Sea Level
FALSE
GPS:GPSAltitude
157.641 m
TRUE
XMP-GPano:StitchingSoftware
Spherical Metadata Tool
TRUE
XMP-GPano:SourcePhotosCount
2
TRUE
XMP-GPano:UsePanoramaViewer
true
TRUE
XMP-GPano:ProjectionType
equirectangular
TRUE
XMP-GPano:CroppedAreaImageHeightPixels
2688
TRUE
XMP-GPano:CroppedAreaImageWidthPixels
5376
TRUE
XMP-GPano:FullPanoHeightPixels
2688
TRUE
XMP-GPano:FullPanoWidthPixels
5376
TRUE
XMP-GPano:CroppedAreaLeftPixels
0
TRUE
XMP-GPano:CroppedAreaTopPixels
0
FALSE
IFD0:Model
GoPro Max

3. Proceed to Final steps

Now final calculations can be preformed. See final processing steps: pro-008e - Final processing steps

pro-008e - Final processing steps

1. Process aggregated metrics for sequence

To support some of the UI views, the following metrics are processed for each photo in Sequence.
Note, some of these values are already calculated during client side validation (e.g. distance to next photo) for images BUT also need to be processed server side for the case of images extracted from video (where the client side checks are not possible).
Measurement
Units
Distance to next photo
meters
Time to next photo
meters
Elevation change to next photo
meters
Heading to next photo
degrees
Speed to next photo
meters/second
Velocity North to next photo
meters/second
Velocity East to next photo
meters/second
Velocity Up to next photo
meters/second

2. Processing complete

Once processing complete all temporary files are then removed from media processing server.
Note, it can take some time for sequence to enter processed state from upload complete. A user can see sequence(s) listed from upload immediately, but cannot modify until final processing complete.
Last modified 2d ago
Copy link
Contents
pro-001u - EAC GoPro Video (.360)
1. Extract metadata
2. Convert to frames
3. Save output to S3
4. Merge frames into equirectangular images
5. Inject metadata (inc telemetry) into each frame
6. Save final jpgs to S3
7. Enter image processing workflow
pro-002u - Stitched equirectangular video (.mp4)
1. Extract metadata
2. Break into frames
3. Save output to S3
4. Inject metadata (inc telemetry) into each frame
5. Save final jpgs to S3
6. Enter image processing workflow
pro-003u - 2 unstitched equirectangular videos (mp4's)
1. Extract metadata from front video
2. Convert both videos to frames
3. Save output to S3
4. Merge frames into equirectangular images
5. Inject metadata (inc telemetry) into each frame
6. Save final jpgs to S3
7. Enter image processing workflow
pro-004u - Hero video (.mp4)
1. Extract metadata
2. Break into frames
3. Save output to S3
4. Inject metadata (inc telemetry) into each frame
5. Save final jpgs to S3
6. Enter image processing workflow
pro-005u - Unstitched equirectangular images (jpg)
1. Extract metadata from all front photos
2. Merge frames into equirectangular images
3. Inject metadata (inc telemetry) into each frame
4. Save final jpgs to S3
5. Enter image processing workflow
pro-006e - HERO (non-equirectangular) images
1. Add watermark
2. Write metadata to DB
3. Proceed to Final steps
pro-007e - Stitched equirectangular images (jpg)
1. Add nadir
2. Write metadata to DB
3. Proceed to Final steps
pro-008e - Final processing steps
1. Process aggregated metrics for sequence
2. Processing complete