Skip to content

Commit

Permalink
feat(android): add contours
Browse files Browse the repository at this point in the history
  • Loading branch information
@trancee
trancee committed Sep 17, 2023
1 parent 2720218 commit f18711b
Show file tree
Hide file tree
Showing 3 changed files with 133 additions and 54 deletions.
39 changes: 30 additions & 9 deletions packages/face-mesh-detection/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -89,11 +89,12 @@ When <a href="#facemesh">`FaceMesh`</a> is selected, <a href="#facemesh">`FaceMe
Each point is represented by <a href="#facemeshpoint">`FaceMeshPoint`</a> describing a specific position in detected face.
The triangle information is a group of 3 `FaceMeshPoint`s representing a valid surface on Face (e.g. a valid small surface on nose tip).

| Prop | Type | Description | Since |
| -------------------- | ------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----- |
| **`bounds`** | <code><a href="#rect">Rect</a></code> | Returns the axis-aligned bounding rectangle of the detected face mesh. | 5.3.0 |
| **`faceMeshPoints`** | <code>FaceMeshPoint[]</code> | Returns a list of <a href="#facemeshpoint">`FaceMeshPoint`</a> representing the whole detected face. | 5.3.0 |
| **`triangles`** | <code>Triangle[]</code> | Returns a list of <a href="#triangle">`Triangle`</a> representing logical triangle surfaces of detected face. Each <a href="#triangle">`Triangle`</a> contains 3 <a href="#facemeshpoint">`FaceMeshPoint`</a>, representing 3 points of the triangle surface. The sequence of the 3 points are constant and always counter clockwise in face mesh. | 5.3.0 |
| Prop | Type | Description | Since |
| -------------------- | --------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----- |
| **`bounds`** | <code><a href="#rect">Rect</a></code> | Returns the axis-aligned bounding rectangle of the detected face mesh. | 5.3.0 |
| **`contours`** | <code><a href="#contours">Contours</a></code> | Returns contours with a list of <a href="#facemeshpoint">`FaceMeshPoint`</a> representing the detected face. | 5.3.0 |
| **`faceMeshPoints`** | <code>FaceMeshPoint[]</code> | Returns a list of <a href="#facemeshpoint">`FaceMeshPoint`</a> representing the whole detected face. | 5.3.0 |
| **`triangles`** | <code>Triangle[]</code> | Returns a list of <a href="#triangle">`Triangle`</a> representing logical triangle surfaces of detected face. Each <a href="#triangle">`Triangle`</a> contains 3 <a href="#facemeshpoint">`FaceMeshPoint`</a>, representing 3 points of the triangle surface. The sequence of the 3 points are constant and always counter clockwise in face mesh. | 5.3.0 |


#### Rect
Expand All @@ -108,6 +109,26 @@ The triangle information is a group of 3 `FaceMeshPoint`s representing a valid s
| **`bottom`** | <code>number</code> | The Y coordinate of the bottom of the rectangle | 5.3.0 |


#### Contours

Represents contours with their face mesh points.

| Prop | Type | Description | Since |
| ------------------------ | ---------------------------- | -------------------------------------------------------- | ----- |
| **`faceOval`** | <code>FaceMeshPoint[]</code> | Returns all points for the `FaceOval` contour. | 5.3.0 |
| **`leftEyebrowTop`** | <code>FaceMeshPoint[]</code> | Returns all points for the `LeftEyebrowTop` contour. | 5.3.0 |
| **`leftEyebrowBottom`** | <code>FaceMeshPoint[]</code> | Returns all points for the `LeftEyebrowBottom` contour. | 5.3.0 |
| **`rightEyebrowTop`** | <code>FaceMeshPoint[]</code> | Returns all points for the `RightEyebrowTop` contour. | 5.3.0 |
| **`rightEyebrowBottom`** | <code>FaceMeshPoint[]</code> | Returns all points for the `RightEyebrowBottom` contour. | 5.3.0 |
| **`leftEye`** | <code>FaceMeshPoint[]</code> | Returns all points for the `LeftEye` contour. | 5.3.0 |
| **`rightEye`** | <code>FaceMeshPoint[]</code> | Returns all points for the `RightEye` contour. | 5.3.0 |
| **`upperLipTop`** | <code>FaceMeshPoint[]</code> | Returns all points for the `UpperLipTop` contour. | 5.3.0 |
| **`upperLipBottom`** | <code>FaceMeshPoint[]</code> | Returns all points for the `UpperLipBottom` contour. | 5.3.0 |
| **`lowerLipTop`** | <code>FaceMeshPoint[]</code> | Returns all points for the `LowerLipTop` contour. | 5.3.0 |
| **`lowerLipBottom`** | <code>FaceMeshPoint[]</code> | Returns all points for the `LowerLipBottom` contour. | 5.3.0 |
| **`noseBridge`** | <code>FaceMeshPoint[]</code> | Returns all points for the `NoseBridge` contour. | 5.3.0 |


#### FaceMeshPoint

Represents a 3D point in face mesh.
Expand All @@ -117,10 +138,10 @@ The index is an unique ID meaning a fixed position on face, ranging from 0 to 46
In <a href="#point3d">`Point3D`</a>, `x` and `y` are pixel location of detected face in `InputImage`.
`z` is also scaled to image size, while the origin will be somewhere in the center of all 468 face mesh points.

| Prop | Type | Description | Since |
| -------------- | ---------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----- |
| **`index`** | <code>number</code> | Gets the index of the face mesh point, ranging from 0 to 467. For each specific point, the index is a constant value. | 5.3.0 |
| **`position`** | <code>Point3D[]</code> | Gets a 3D point in face mesh. Inside <a href="#point3d">`Point3D`</a>, `X` and `Y` means a 2D position in original image. More information on the `Z` value: - The unit of measure for the `Z` value is the same as `X` and `Y`. - The smaller the `Z` value, the closer that landmark is to the camera. - The `Z` origin is approximately at the center of all 468 face mesh points. `Z` value will be negative if the point is close to camera and will be positive if the point is away from the camera. | 5.3.0 |
| Prop | Type | Description | Since |
| ----------- | ------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----- |
| **`index`** | <code>number</code> | Gets the index of the face mesh point, ranging from 0 to 467. For each specific point, the index is a constant value. | 5.3.0 |
| **`point`** | <code><a href="#point3d">Point3D</a></code> | Gets a 3D point in face mesh. Inside <a href="#point3d">`Point3D`</a>, `X` and `Y` means a 2D position in original image. More information on the `Z` value: - The unit of measure for the `Z` value is the same as `X` and `Y`. - The smaller the `Z` value, the closer that landmark is to the camera. - The `Z` origin is approximately at the center of all 468 face mesh points. `Z` value will be negative if the point is close to camera and will be positive if the point is away from the camera. | 5.3.0 |


#### Point3D
Expand Down
86 changes: 69 additions & 17 deletions ...io/capawesome/capacitorjs/plugins/mlkit/facemeshdetection/classes/ProcessImageResult.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,22 @@ public class ProcessImageResult {

private final List<FaceMesh> faceMeshs;

@FaceMesh.ContourType
private final int[] contourTypes = {
FaceMesh.FACE_OVAL,
FaceMesh.LEFT_EYEBROW_TOP,
FaceMesh.LEFT_EYEBROW_BOTTOM,
FaceMesh.RIGHT_EYEBROW_TOP,
FaceMesh.RIGHT_EYEBROW_BOTTOM,
FaceMesh.LEFT_EYE,
FaceMesh.RIGHT_EYE,
FaceMesh.UPPER_LIP_TOP,
FaceMesh.UPPER_LIP_BOTTOM,
FaceMesh.LOWER_LIP_TOP,
FaceMesh.LOWER_LIP_BOTTOM,
FaceMesh.NOSE_BRIDGE
};

public ProcessImageResult(List<FaceMesh> faceMeshs) {
this.faceMeshs = faceMeshs;
}
Expand Down Expand Up @@ -41,6 +57,26 @@ private JSObject createFaceMeshResult(FaceMesh faceMesh) {
JSObject boundsResult = this.createBoundsResult(boundingBox);
result.put("bounds", boundsResult);

JSObject contoursResult = new JSObject();
for (int contourType : contourTypes) {
JSArray faceMeshPointsResult = new JSArray();
List<FaceMeshPoint> faceMeshPoints = faceMesh.getPoints(contourType);
for (FaceMeshPoint faceMeshPoint : faceMeshPoints) {
int index = faceMeshPoint.getIndex();
PointF3D position = faceMeshPoint.getPosition();

JSObject faceMeshPointResult = this.createFaceMeshPointResult(index, position);
faceMeshPointsResult.put(faceMeshPointResult);
}

if (faceMeshPointsResult.length() > 0) {
contoursResult.put(contourType(contourType), faceMeshPointsResult);
}
}
if (contoursResult.length() > 0) {
result.put("contours", contoursResult);
}

JSArray faceMeshPointsResult = new JSArray();
List<FaceMeshPoint> faceMeshPoints = faceMesh.getAllPoints();
for (FaceMeshPoint faceMeshPoint : faceMeshPoints) {
Expand All @@ -54,15 +90,6 @@ private JSObject createFaceMeshResult(FaceMesh faceMesh) {
result.put("faceMeshPoints", faceMeshPointsResult);
}

// JSArray contoursResult = new JSArray();
// if (pointsResult.length() > 0) {
// JSObject contourResult = this.createContourResult(contour, pointsResult);
// contoursResult.put(contourResult);
// }
// if (contoursResult.length() > 0) {
// result.put("contours", contoursResult);
// }

JSArray trianglesResult = new JSArray();
List<Triangle<FaceMeshPoint>> triangles = faceMesh.getAllTriangles();
for (Triangle<FaceMeshPoint> triangle : triangles) {
Expand Down Expand Up @@ -102,21 +129,46 @@ private JSObject createBoundsResult(Rect boundingBox) {
private JSObject createFaceMeshPointResult(int index, PointF3D point) {
JSObject result = new JSObject();
result.put("index", index);
result.put("position", createPositionResult(point));
result.put("point", createPointResult(point));
return result;
}

private JSObject createPositionResult(PointF3D point) {
private JSObject createPointResult(PointF3D point) {
JSObject result = new JSObject();
result.put("x", point.getX());
result.put("y", point.getY());
result.put("z", point.getZ());
return result;
}
// private JSObject createContourResult(FaceContour contour, JSArray pointsResult) {
// JSObject result = new JSObject();
// result.put("type", contour.getFaceContourType());
// result.put("points", pointsResult);
// return result;
// }

private String contourType(@FaceMesh.ContourType int contourType) {
switch (contourType) {
case FaceMesh.FACE_OVAL:
return "faceOval";
case FaceMesh.LEFT_EYEBROW_TOP:
return "leftEyebrowTop";
case FaceMesh.LEFT_EYEBROW_BOTTOM:
return "leftEyebrowBottom";
case FaceMesh.RIGHT_EYEBROW_TOP:
return "rightEyebrowTop";
case FaceMesh.RIGHT_EYEBROW_BOTTOM:
return "rightEyebrowBottom";
case FaceMesh.LEFT_EYE:
return "leftEye";
case FaceMesh.RIGHT_EYE:
return "rightEye";
case FaceMesh.UPPER_LIP_TOP:
return "upperLipTop";
case FaceMesh.UPPER_LIP_BOTTOM:
return "upperLipBottom";
case FaceMesh.LOWER_LIP_TOP:
return "lowerLipTop";
case FaceMesh.LOWER_LIP_BOTTOM:
return "lowerLipBottom";
case FaceMesh.NOSE_BRIDGE:
return "noseBridge";
default:
return "";
}
}
}
62 changes: 34 additions & 28 deletions packages/face-mesh-detection/src/definitions.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -91,6 +91,13 @@ export interface FaceMesh {
*/
bounds: Rect;

/**
* Returns contours with a list of `FaceMeshPoint` representing the detected face.
*
* @since 5.3.0
*/
contours?: Contours;

/**
* Returns a list of `FaceMeshPoint` representing the whole detected face.
*
Expand Down Expand Up @@ -173,7 +180,7 @@ export interface FaceMeshPoint {
*
* @since 5.3.0
*/
position: Point3D[];
point: Point3D;
}

/**
Expand Down Expand Up @@ -217,82 +224,81 @@ export interface Triangle {
}

/**
* Type of face contour.
* Represents contours with their face mesh points.
*
* @since 5.3.0
* @see https://developers.google.com/android/reference/com/google/mlkit/vision/facemesh/FaceMesh.ContourType
*/
export enum ContourType {
export interface Contours {
/**
* The outline of the subject's face.
* Returns all points for the `FaceOval` contour.
*
* @since 5.3.0
*/
FaceOval = 1,
faceOval?: FaceMeshPoint[];
/**
* The top outline of the subject's left eyebrow.
* Returns all points for the `LeftEyebrowTop` contour.
*
* @since 5.3.0
*/
LeftEyebrowTop = 2,
leftEyebrowTop?: FaceMeshPoint[];
/**
* The bottom outline of the subject's left eyebrow.
* Returns all points for the `LeftEyebrowBottom` contour.
*
* @since 5.3.0
*/
LeftEyebrowBottom = 3,
leftEyebrowBottom?: FaceMeshPoint[];
/**
* The top outline of the subject's right eyebrow.
* Returns all points for the `RightEyebrowTop` contour.
*
* @since 5.3.0
*/
RightEyebrowTop = 4,
rightEyebrowTop?: FaceMeshPoint[];
/**
* The bottom outline of the subject's right eyebrow.
* Returns all points for the `RightEyebrowBottom` contour.
*
* @since 5.3.0
*/
RightEyebrowBottom = 5,
rightEyebrowBottom?: FaceMeshPoint[];
/**
* The outline of the subject's left eye cavity.
* Returns all points for the `LeftEye` contour.
*
* @since 5.3.0
*/
LeftEye = 6,
leftEye?: FaceMeshPoint[];
/**
* The outline of the subject's right eye cavity.
* Returns all points for the `RightEye` contour.
*
* @since 5.3.0
*/
RightEye = 7,
rightEye?: FaceMeshPoint[];
/**
* The top outline of the subject's upper lip.
* Returns all points for the `UpperLipTop` contour.
*
* @since 5.3.0
*/
UpperLipTop = 8,
upperLipTop?: FaceMeshPoint[];
/**
* The bottom outline of the subject's upper lip.
* Returns all points for the `UpperLipBottom` contour.
*
* @since 5.3.0
*/
UpperLipBottom = 9,
upperLipBottom?: FaceMeshPoint[];
/**
* The top outline of the subject's lower lip.
* Returns all points for the `LowerLipTop` contour.
*
* @since 5.3.0
*/
LowerLipTop = 10,
lowerLipTop?: FaceMeshPoint[];
/**
* The bottom outline of the subject's lower lip.
* Returns all points for the `LowerLipBottom` contour.
*
* @since 5.3.0
*/
LowerLipBottom = 11,
lowerLipBottom?: FaceMeshPoint[];
/**
* The outline of the subject's nose bridge.
* Returns all points for the `NoseBridge` contour.
*
* @since 5.3.0
*/
NoseBridge = 12,
noseBridge?: FaceMeshPoint[];
}

0 comments on commit f18711b

Please sign in to comment.