ARKit materials have a .blendMode property that allows you to overlay the color of your rendered object on top of the video image. This works great, however, I need to use a different blend mode to apply the color to the video image. I have tried creating a custom SCNTechnique that applies my custom blend mode, and the shader works, but in order to blend the render objects color with the original ARKit video feed, I need access to the image before rendering. I have this working with the mask created from people occlusion in Metal rendering, but not with my own geometry. Additionally, I'd prefer to keep things in ARKit with the normal SceneKit rendering. Does anyone know how I can access the original ARKit video images in a custom SCNTechnique shader within the bounds of SceneKit? How I imagine this working is as part of the SCNTechnique Passes, I render the original video to a texture or pass it in somehow, then render the geometry to a different texture, then on a third pass, use both of those textures to compute my final image.
How to access the original ARKit camera image in a custom shader or SCNTechnique?
186 views Asked by Davido At
0
There are 0 answers
Related Questions in SWIFT
- Navigate after logged in with webservice
- URLSession requesting JSON array from server not working
- When using onDrag in SwiftUI on Mac how can I detect when the dragged object has been released anywhere?
- Protect OpenAI key using Firebase function
- How to correct error: "Cannot convert value of type 'MyType.Type' to expected argument type 'Binding<MyType>'"?
- How to share metadata of an audio url file to a WhatsApp conversation with friends
- Using @Bindable with a Observable type in SwiftUI
- How to make a scroll view of 9 images in a forEach loop open on image 6 if image 6 is clicked on from a grid?
- Using MTLPixelFormat.rgba16Float results in random round-off errors
- Search and highlight text of current text in PDFKit Swift
- How is passing a function as a parameter related to escaping autoclosure?
- Actionable notification api call not working in background
- Custom layout occupies all horizontal space
- Is it possible to fix slow CKAsset loading on Cloudkit?
- Thread 1: Fatal error: Unexpectedly found nil while implicitly unwrapping an Optional value - MapView.isMyLocationEnabled
Related Questions in SCENEKIT
- Create and customize T-shirt using SceneKit
- Multiple SCNActions
- How to Make Xcode Recognize Morph Targets in a DAE File Imported from Blender?
- ARKit combining ARImageAnchor position with offset values
- Fetching the animations added to a SCNNode
- RealityKit: VideoMaterial causing odd outline around transparent video?
- Can't add a SCNNode to the rootNode
- How to screenshot a SCNScene or SceneView?
- Issues with Vertical Stretching and Horizontal Movement in Metal Shading Language Ray Tracing Shader
- Why does AVPlayer doesn't orient video correctly when using SceneKit?
- How to load 3D model to UIView with better quality in iOS Swift?
- Several instances of 3D model with skinner, and duplication of weights/indices information
- SceneKit physicBody angularVelocity as local "body fixed" angular rates
- How to draw 3D circles/ring in RealityKit?
- SCNSkinner.boneIndices access generates GBytes of memory allocation - why?
Related Questions in ARKIT
- UE4: CodeSign Failed, get-task-allow entitlement doesn't match
- Is it possible to use Apple's ARKit to measure the length and width of an individual's foot to estimate shoe size?
- How can I detect plane and add model entity on it in visionOS?
- AR Scene Black Screen Issue in Android with AR Core AR Foundation and Niantic ARDK 3
- RealityKit - Add a line from one Model entity to center of screen as it moves
- Getting Call to main actor-isolated static method 'loadMainScene()' in a synchronous nonisolated context error in ARKit swiftui application
- Why custom created entity does not rotate with BodyAnchor's rotation?
- How to I wrap a UIKit view in SwiftUI with a delegate callback requiring a return value?
- How to Make Xcode Recognize Morph Targets in a DAE File Imported from Blender?
- ARKit not perfectly locking in user heading
- ARKit combining ARImageAnchor position with offset values
- how to get the bouding box dimension when using the object capture api in iOS17 after detecting the object
- ARKit Body Tracking head-body rotation inaccuracy when model turn back
- Unable to present an Immersive Space for id: Error Domain=FBSWorkspaceErrorDomain Code=1 "scene invalidated before create completion"
- RealityKit: VideoMaterial causing odd outline around transparent video?
Related Questions in METAL
- Using MTLPixelFormat.rgba16Float results in random round-off errors
- The Three-Image Problem: How do I set tagged color data in MTKView and CIContext to properly display all three images?
- Max total threads per thread group and gpu cores on apple machines
- MTLBuffers returning "shifted" values in kernel shader using structs, works on simulator, not on device
- Failed to install llama-cpp-python with Metal on M2 Ultra
- How to use multiple mps devices in M1 Mac for PyTorch
- Metal: Image blocks and MSAA
- Basic ImGui program doesn't render, leaks memory and crashes
- How to set frame rate in GLFW based metal application?
- In Metal, how to wait for all GPU operations in my process to complete?
- Metal sampling texture from argument buffer generates GPU Address Fault
- glDrawPixels equalent for Metal API
- Lack of non-normalized 10_10_10_2 vertex format in Metal
- Device orientation dependent SwiftUI Metal problem: RBLayer: unable to create texture: BGRA8Unorm
- How to render content of CALayer and sync with it in real-time
Related Questions in SCNTECHNIQUE
- How do I get a SceneKit SCNTechnique to work with includeCategoryBitmask?
- writing to a depth target not working when using SCNTechnique with a metal shader
- How to stop Swift ARKit app from reverting SCN Technique, and world tracking issues
- SCNProgram read from depth buffer
- How to access the original ARKit camera image in a custom shader or SCNTechnique?
- How to add "excludeCategoryMask" for two different Categories in the same pass?
- Metal / SceneKit Fragment Shaders - How to avoid rendering ontop of other geometry?
- How add custom SCNTechnique pass to specific area, without messing with the rest of Sceneview fullscreen area?
- Is it possible to use CIFilters directly in Metal?
- Modify Metal fragment shading based on vertex world position Y coordinate
- SCNTechnique on ARSCNView does not affect CameraFeed (Scene Background) on iOS 13
- How come SCNTechnique doesn't have a .library property like SCNProgram?
- Why does my scene camera only show black screen when I use SCNTechnique?
- iOS SceneKit SCNTechnique Metal vertex and fragment methods not work in iOS12
- Blank depth buffer from ARSCNView in DRAW_QUAD pass of SCNTechnique
Popular Questions
- How do I undo the most recent local commits in Git?
- How can I remove a specific item from an array in JavaScript?
- How do I delete a Git branch locally and remotely?
- Find all files containing a specific text (string) on Linux?
- How do I revert a Git repository to a previous commit?
- How do I create an HTML button that acts like a link?
- How do I check out a remote Git branch?
- How do I force "git pull" to overwrite local files?
- How do I list all files of a directory?
- How to check whether a string contains a substring in JavaScript?
- How do I redirect to another webpage?
- How can I iterate over rows in a Pandas DataFrame?
- How do I convert a String to an int in Java?
- Does Python have a string 'contains' substring method?
- How do I check if a string contains a specific word?
Trending Questions
- UIImageView Frame Doesn't Reflect Constraints
- Is it possible to use adb commands to click on a view by finding its ID?
- How to create a new web character symbol recognizable by html/javascript?
- Why isn't my CSS3 animation smooth in Google Chrome (but very smooth on other browsers)?
- Heap Gives Page Fault
- Connect ffmpeg to Visual Studio 2008
- Both Object- and ValueAnimator jumps when Duration is set above API LvL 24
- How to avoid default initialization of objects in std::vector?
- second argument of the command line arguments in a format other than char** argv or char* argv[]
- How to improve efficiency of algorithm which generates next lexicographic permutation?
- Navigating to the another actvity app getting crash in android
- How to read the particular message format in android and store in sqlite database?
- Resetting inventory status after order is cancelled
- Efficiently compute powers of X in SSE/AVX
- Insert into an external database using ajax and php : POST 500 (Internal Server Error)