diff --git a/packages/face-mesh-detection/README.md b/packages/face-mesh-detection/README.md index fced07e..c536d21 100644 --- a/packages/face-mesh-detection/README.md +++ b/packages/face-mesh-detection/README.md @@ -89,11 +89,12 @@ When `FaceMesh` is selected, `FaceMe Each point is represented by `FaceMeshPoint` describing a specific position in detected face. The triangle information is a group of 3 `FaceMeshPoint`s representing a valid surface on Face (e.g. a valid small surface on nose tip). -| Prop | Type | Description | Since | -| -------------------- | ------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----- | -| **`bounds`** | Rect | Returns the axis-aligned bounding rectangle of the detected face mesh. | 5.3.0 | -| **`faceMeshPoints`** | FaceMeshPoint[] | Returns a list of `FaceMeshPoint` representing the whole detected face. | 5.3.0 | -| **`triangles`** | Triangle[] | Returns a list of `Triangle` representing logical triangle surfaces of detected face. Each `Triangle` contains 3 `FaceMeshPoint`, representing 3 points of the triangle surface. The sequence of the 3 points are constant and always counter clockwise in face mesh. | 5.3.0 | +| Prop | Type | Description | Since | +| -------------------- | --------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----- | +| **`bounds`** | Rect | Returns the axis-aligned bounding rectangle of the detected face mesh. | 5.3.0 | +| **`contours`** | Contours | Returns contours with a list of `FaceMeshPoint` representing the detected face. | 5.3.0 | +| **`faceMeshPoints`** | FaceMeshPoint[] | Returns a list of `FaceMeshPoint` representing the whole detected face. | 5.3.0 | +| **`triangles`** | Triangle[] | Returns a list of `Triangle` representing logical triangle surfaces of detected face. Each `Triangle` contains 3 `FaceMeshPoint`, representing 3 points of the triangle surface. The sequence of the 3 points are constant and always counter clockwise in face mesh. | 5.3.0 | #### Rect @@ -108,6 +109,26 @@ The triangle information is a group of 3 `FaceMeshPoint`s representing a valid s | **`bottom`** | number | The Y coordinate of the bottom of the rectangle | 5.3.0 | +#### Contours + +Represents contours with their face mesh points. + +| Prop | Type | Description | Since | +| ------------------------ | ---------------------------- | -------------------------------------------------------- | ----- | +| **`faceOval`** | FaceMeshPoint[] | Returns all points for the `FaceOval` contour. | 5.3.0 | +| **`leftEyebrowTop`** | FaceMeshPoint[] | Returns all points for the `LeftEyebrowTop` contour. | 5.3.0 | +| **`leftEyebrowBottom`** | FaceMeshPoint[] | Returns all points for the `LeftEyebrowBottom` contour. | 5.3.0 | +| **`rightEyebrowTop`** | FaceMeshPoint[] | Returns all points for the `RightEyebrowTop` contour. | 5.3.0 | +| **`rightEyebrowBottom`** | FaceMeshPoint[] | Returns all points for the `RightEyebrowBottom` contour. | 5.3.0 | +| **`leftEye`** | FaceMeshPoint[] | Returns all points for the `LeftEye` contour. | 5.3.0 | +| **`rightEye`** | FaceMeshPoint[] | Returns all points for the `RightEye` contour. | 5.3.0 | +| **`upperLipTop`** | FaceMeshPoint[] | Returns all points for the `UpperLipTop` contour. | 5.3.0 | +| **`upperLipBottom`** | FaceMeshPoint[] | Returns all points for the `UpperLipBottom` contour. | 5.3.0 | +| **`lowerLipTop`** | FaceMeshPoint[] | Returns all points for the `LowerLipTop` contour. | 5.3.0 | +| **`lowerLipBottom`** | FaceMeshPoint[] | Returns all points for the `LowerLipBottom` contour. | 5.3.0 | +| **`noseBridge`** | FaceMeshPoint[] | Returns all points for the `NoseBridge` contour. | 5.3.0 | + + #### FaceMeshPoint Represents a 3D point in face mesh. @@ -117,10 +138,10 @@ The index is an unique ID meaning a fixed position on face, ranging from 0 to 46 In `Point3D`, `x` and `y` are pixel location of detected face in `InputImage`. `z` is also scaled to image size, while the origin will be somewhere in the center of all 468 face mesh points. -| Prop | Type | Description | Since | -| -------------- | ---------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----- | -| **`index`** | number | Gets the index of the face mesh point, ranging from 0 to 467. For each specific point, the index is a constant value. | 5.3.0 | -| **`position`** | Point3D[] | Gets a 3D point in face mesh. Inside `Point3D`, `X` and `Y` means a 2D position in original image. More information on the `Z` value: - The unit of measure for the `Z` value is the same as `X` and `Y`. - The smaller the `Z` value, the closer that landmark is to the camera. - The `Z` origin is approximately at the center of all 468 face mesh points. `Z` value will be negative if the point is close to camera and will be positive if the point is away from the camera. | 5.3.0 | +| Prop | Type | Description | Since | +| ----------- | ------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----- | +| **`index`** | number | Gets the index of the face mesh point, ranging from 0 to 467. For each specific point, the index is a constant value. | 5.3.0 | +| **`point`** | Point3D | Gets a 3D point in face mesh. Inside `Point3D`, `X` and `Y` means a 2D position in original image. More information on the `Z` value: - The unit of measure for the `Z` value is the same as `X` and `Y`. - The smaller the `Z` value, the closer that landmark is to the camera. - The `Z` origin is approximately at the center of all 468 face mesh points. `Z` value will be negative if the point is close to camera and will be positive if the point is away from the camera. | 5.3.0 | #### Point3D diff --git a/packages/face-mesh-detection/android/src/main/java/io/capawesome/capacitorjs/plugins/mlkit/facemeshdetection/classes/ProcessImageResult.java b/packages/face-mesh-detection/android/src/main/java/io/capawesome/capacitorjs/plugins/mlkit/facemeshdetection/classes/ProcessImageResult.java index a941636..d3414ef 100644 --- a/packages/face-mesh-detection/android/src/main/java/io/capawesome/capacitorjs/plugins/mlkit/facemeshdetection/classes/ProcessImageResult.java +++ b/packages/face-mesh-detection/android/src/main/java/io/capawesome/capacitorjs/plugins/mlkit/facemeshdetection/classes/ProcessImageResult.java @@ -13,6 +13,22 @@ public class ProcessImageResult { private final List faceMeshs; + @FaceMesh.ContourType + private final int[] contourTypes = { + FaceMesh.FACE_OVAL, + FaceMesh.LEFT_EYEBROW_TOP, + FaceMesh.LEFT_EYEBROW_BOTTOM, + FaceMesh.RIGHT_EYEBROW_TOP, + FaceMesh.RIGHT_EYEBROW_BOTTOM, + FaceMesh.LEFT_EYE, + FaceMesh.RIGHT_EYE, + FaceMesh.UPPER_LIP_TOP, + FaceMesh.UPPER_LIP_BOTTOM, + FaceMesh.LOWER_LIP_TOP, + FaceMesh.LOWER_LIP_BOTTOM, + FaceMesh.NOSE_BRIDGE + }; + public ProcessImageResult(List faceMeshs) { this.faceMeshs = faceMeshs; } @@ -41,6 +57,26 @@ private JSObject createFaceMeshResult(FaceMesh faceMesh) { JSObject boundsResult = this.createBoundsResult(boundingBox); result.put("bounds", boundsResult); + JSObject contoursResult = new JSObject(); + for (int contourType : contourTypes) { + JSArray faceMeshPointsResult = new JSArray(); + List faceMeshPoints = faceMesh.getPoints(contourType); + for (FaceMeshPoint faceMeshPoint : faceMeshPoints) { + int index = faceMeshPoint.getIndex(); + PointF3D position = faceMeshPoint.getPosition(); + + JSObject faceMeshPointResult = this.createFaceMeshPointResult(index, position); + faceMeshPointsResult.put(faceMeshPointResult); + } + + if (faceMeshPointsResult.length() > 0) { + contoursResult.put(contourType(contourType), faceMeshPointsResult); + } + } + if (contoursResult.length() > 0) { + result.put("contours", contoursResult); + } + JSArray faceMeshPointsResult = new JSArray(); List faceMeshPoints = faceMesh.getAllPoints(); for (FaceMeshPoint faceMeshPoint : faceMeshPoints) { @@ -54,15 +90,6 @@ private JSObject createFaceMeshResult(FaceMesh faceMesh) { result.put("faceMeshPoints", faceMeshPointsResult); } - // JSArray contoursResult = new JSArray(); - // if (pointsResult.length() > 0) { - // JSObject contourResult = this.createContourResult(contour, pointsResult); - // contoursResult.put(contourResult); - // } - // if (contoursResult.length() > 0) { - // result.put("contours", contoursResult); - // } - JSArray trianglesResult = new JSArray(); List> triangles = faceMesh.getAllTriangles(); for (Triangle triangle : triangles) { @@ -102,21 +129,46 @@ private JSObject createBoundsResult(Rect boundingBox) { private JSObject createFaceMeshPointResult(int index, PointF3D point) { JSObject result = new JSObject(); result.put("index", index); - result.put("position", createPositionResult(point)); + result.put("point", createPointResult(point)); return result; } - private JSObject createPositionResult(PointF3D point) { + private JSObject createPointResult(PointF3D point) { JSObject result = new JSObject(); result.put("x", point.getX()); result.put("y", point.getY()); result.put("z", point.getZ()); return result; } - // private JSObject createContourResult(FaceContour contour, JSArray pointsResult) { - // JSObject result = new JSObject(); - // result.put("type", contour.getFaceContourType()); - // result.put("points", pointsResult); - // return result; - // } + + private String contourType(@FaceMesh.ContourType int contourType) { + switch (contourType) { + case FaceMesh.FACE_OVAL: + return "faceOval"; + case FaceMesh.LEFT_EYEBROW_TOP: + return "leftEyebrowTop"; + case FaceMesh.LEFT_EYEBROW_BOTTOM: + return "leftEyebrowBottom"; + case FaceMesh.RIGHT_EYEBROW_TOP: + return "rightEyebrowTop"; + case FaceMesh.RIGHT_EYEBROW_BOTTOM: + return "rightEyebrowBottom"; + case FaceMesh.LEFT_EYE: + return "leftEye"; + case FaceMesh.RIGHT_EYE: + return "rightEye"; + case FaceMesh.UPPER_LIP_TOP: + return "upperLipTop"; + case FaceMesh.UPPER_LIP_BOTTOM: + return "upperLipBottom"; + case FaceMesh.LOWER_LIP_TOP: + return "lowerLipTop"; + case FaceMesh.LOWER_LIP_BOTTOM: + return "lowerLipBottom"; + case FaceMesh.NOSE_BRIDGE: + return "noseBridge"; + default: + return ""; + } + } } diff --git a/packages/face-mesh-detection/src/definitions.ts b/packages/face-mesh-detection/src/definitions.ts index 8dcb9a0..6823905 100644 --- a/packages/face-mesh-detection/src/definitions.ts +++ b/packages/face-mesh-detection/src/definitions.ts @@ -91,6 +91,13 @@ export interface FaceMesh { */ bounds: Rect; + /** + * Returns contours with a list of `FaceMeshPoint` representing the detected face. + * + * @since 5.3.0 + */ + contours?: Contours; + /** * Returns a list of `FaceMeshPoint` representing the whole detected face. * @@ -173,7 +180,7 @@ export interface FaceMeshPoint { * * @since 5.3.0 */ - position: Point3D[]; + point: Point3D; } /** @@ -217,82 +224,81 @@ export interface Triangle { } /** - * Type of face contour. + * Represents contours with their face mesh points. * * @since 5.3.0 - * @see https://developers.google.com/android/reference/com/google/mlkit/vision/facemesh/FaceMesh.ContourType */ -export enum ContourType { +export interface Contours { /** - * The outline of the subject's face. + * Returns all points for the `FaceOval` contour. * * @since 5.3.0 */ - FaceOval = 1, + faceOval?: FaceMeshPoint[]; /** - * The top outline of the subject's left eyebrow. + * Returns all points for the `LeftEyebrowTop` contour. * * @since 5.3.0 */ - LeftEyebrowTop = 2, + leftEyebrowTop?: FaceMeshPoint[]; /** - * The bottom outline of the subject's left eyebrow. + * Returns all points for the `LeftEyebrowBottom` contour. * * @since 5.3.0 */ - LeftEyebrowBottom = 3, + leftEyebrowBottom?: FaceMeshPoint[]; /** - * The top outline of the subject's right eyebrow. + * Returns all points for the `RightEyebrowTop` contour. * * @since 5.3.0 */ - RightEyebrowTop = 4, + rightEyebrowTop?: FaceMeshPoint[]; /** - * The bottom outline of the subject's right eyebrow. + * Returns all points for the `RightEyebrowBottom` contour. * * @since 5.3.0 */ - RightEyebrowBottom = 5, + rightEyebrowBottom?: FaceMeshPoint[]; /** - * The outline of the subject's left eye cavity. + * Returns all points for the `LeftEye` contour. * * @since 5.3.0 */ - LeftEye = 6, + leftEye?: FaceMeshPoint[]; /** - * The outline of the subject's right eye cavity. + * Returns all points for the `RightEye` contour. * * @since 5.3.0 */ - RightEye = 7, + rightEye?: FaceMeshPoint[]; /** - * The top outline of the subject's upper lip. + * Returns all points for the `UpperLipTop` contour. * * @since 5.3.0 */ - UpperLipTop = 8, + upperLipTop?: FaceMeshPoint[]; /** - * The bottom outline of the subject's upper lip. + * Returns all points for the `UpperLipBottom` contour. * * @since 5.3.0 */ - UpperLipBottom = 9, + upperLipBottom?: FaceMeshPoint[]; /** - * The top outline of the subject's lower lip. + * Returns all points for the `LowerLipTop` contour. * * @since 5.3.0 */ - LowerLipTop = 10, + lowerLipTop?: FaceMeshPoint[]; /** - * The bottom outline of the subject's lower lip. + * Returns all points for the `LowerLipBottom` contour. * * @since 5.3.0 */ - LowerLipBottom = 11, + lowerLipBottom?: FaceMeshPoint[]; /** - * The outline of the subject's nose bridge. + * Returns all points for the `NoseBridge` contour. * * @since 5.3.0 */ - NoseBridge = 12, + noseBridge?: FaceMeshPoint[]; }