gitlab-com-seaweedfs/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-01-19 02:48:24 +00:00
Code Issues Projects Releases Wiki Activity
8791 commits 28 branches 278 tags 135 MiB
Commit graph

1392 commits

Author SHA1 Message Date
chrislu e024586ff1 adjust conditions 2022-06-15 23:19:39 -07:00
Chris Lu 82f3bcc65e
Merge pull request #3081 from paochiang/volume_upload_limit_fix 
exclude replication from the concurrentUploadLimitMB
 2022-06-15 23:16:44 -07:00
liubaojiang df0ce31a2e add condition when inFlightUploadDataLimitCond signal 2022-06-16 14:07:11 +08:00
liubaojiang 3076ac101e move vs.concurrentUploadLimit != 0 out of the lock 2022-06-16 09:58:44 +08:00
Chris Lu 55b9f281d1
Merge pull request #3185 from kmlebedev/logging_err_process_range_req 2022-06-15 06:33:33 -07:00
Konstantin Lebedev 31a929f7f2 logging processRangeRequest errors 2022-06-15 15:36:43 +05:00
Chris Lu 4eeeb5f50d
Merge pull request #3180 from blacktear23/issue-3149 2022-06-15 00:55:20 -07:00
zhihao.qu 42d04c581b feat(filer.sync): add metricsServer in filer.sync. 
Metrics include:
(1) the offset of the filer.sync
(2) the last send timestamp of the filer subscription
 2022-06-15 11:33:18 +08:00
yulai.li 28cda854eb Add filer command line parameter to let Filer UI show/hide directory delete button 2022-06-15 00:30:49 +08:00
chrislu d12f431d98 collect volume server status 2022-06-12 11:56:23 -07:00
chrislu 444ac21050 go fmt 2022-06-11 09:51:11 -07:00
Chris Lu 0a8b3948db
Merge pull request #3165 from blacktear23/reload-bug-fix 2022-06-10 07:47:43 -07:00
yulai.li 83f8cdd5cb Bug Fix: force reload filer page when upload/delete/rename/create directory 2022-06-10 22:39:21 +08:00
guol-fnst eabf4d4e64 fix typo 2022-06-10 22:03:11 +08:00
chrislu 9f8b72a54d Revert "Merge pull request #3159 from shichanglin5/_duplicateUUID" 
This reverts commit 37da689319, reversing
changes made to 00d53c34c4.
 2022-06-10 06:38:17 -07:00
Chris Lu 37da689319
Merge pull request #3159 from shichanglin5/_duplicateUUID 
perf: Optimized volume handling duplicateUUID logic to avoid quitting…
 2022-06-09 18:01:02 -07:00
shichanglin5 f5b0c04b14 perf: Optimized volume handling duplicateUUID logic to avoid quitting when volume is actualy normal 
Under normal circumstances, there will be no problems, but when the
master is debugged in the local environment, the volume client cannot
communicate with the master normally, so the sendHeartBeat logic is
restarted, and a new connection is created to report the heartbeat. If
the master has not cleared the uuid of the volume at this time, then The
master will respond to volume duplicateUUIDS, and the volume service
will exit, but in fact the uuid of the volume is not duplicated
 2022-06-09 20:41:16 +08:00
chrislu 4fd5f96598 filer: remove replication, collection, disk_type info from entry metadata 
these metadata can change and are not used
 2022-06-06 00:39:35 -07:00
chrislu 8402af0882 filer: detect ttl based on fs configure setting 2022-06-06 00:15:37 -07:00
chrislu ecef844dfc stream read large files 2022-06-05 11:54:04 -07:00
chrislu 48cca4e54f remove unused logic 2022-06-04 19:11:06 -07:00
chrislu 376b005ad3 fix wrong logic 2022-05-31 11:55:58 -07:00
chrislu 27732ecfa4 move s3 related constants from package http to s3_constants 2022-05-30 22:57:41 -07:00
chrislu f4a6da6cb2 Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-05-30 22:47:29 -07:00
chrislu 596c3860ca use final destination to resolve fs configuration 
related to https://github.com/chrislusf/seaweedfs/issues/3075
 2022-05-30 22:47:26 -07:00
chrislu 6adc42147f fresh filer store bootstrap from the oldest peer 2022-05-30 21:27:48 -07:00
liubaojiang f0ee3e6f21 reduce the scope of inFlightUploadDataLimitCond lock 2022-05-31 09:49:07 +08:00
chrislu c59068d0f3 refactor 2022-05-30 16:28:36 -07:00
chrislu 682382648e collect cluster node start time 2022-05-30 16:23:52 -07:00
chrislu f214dfb1f5 stop when in memory log is done 2022-05-30 15:25:21 -07:00
chrislu aece35a64f stop when on disk log is done 2022-05-30 15:20:51 -07:00
chrislu a2b101a737 subscribe metadata between a range 2022-05-30 15:04:19 -07:00
chrislu ca16fbf0ef shell: cluster.ps display filer group 2022-05-29 16:37:14 -07:00
chrislu 7e25a2d416 reduce busy waiting when reading metadata logs 2022-05-24 00:23:53 -07:00
liubaojiang 076e48a676 add inflight upload data wait timeout 2022-05-21 10:38:08 +08:00
liubaojiang 71b2e6223e add inFlightUploadDataLimitCond signal 2022-05-20 15:27:06 +08:00
liubaojiang 1a41691b4c exclude replication from the concurrentUploadLimitMB 2022-05-20 14:40:05 +08:00
Chris Lu cb64983c6b
Merge pull request #3079 from ztinpn/hotfix_fsync 2022-05-19 10:33:42 -07:00
tianzhang 66747ee9c9 hotfix_fsync fix fsync 2022-05-19 23:17:17 +08:00
guol-fnst b12944f9c6 fix naming convention 
notify volume server of duplicate directoris
improve searching efficiency
 2022-05-17 15:41:49 +08:00
guol-fnst 076595fbdd just exit in case of duplicated volume directories were loaded 2022-05-17 15:41:49 +08:00
guol-fnst 8fab39e775 rename UUID file 
fix typo
move locationUUID  into DiskLocation
 2022-05-17 15:41:47 +08:00
guol-fnst de6aa9cce8 avoid duplicated volume directory 2022-05-16 19:33:51 +08:00
Chris Lu a69bf53405
Merge pull request #2996 from kmlebedev/ydb 
ydb filer store
 2022-05-05 15:34:37 -07:00
chrislu b201edb9df fix wrong assignment 2022-05-03 07:12:33 -07:00
chrislu 9271866d1e fix segmentation violation 
fix https://github.com/chrislusf/seaweedfs/issues/3000
 2022-05-02 23:56:11 -07:00
Konstantin Lebedev d8925b4e83 Merge branch 'new_master' into ydb 
# Conflicts:
#	go.mod
#	go.sum
 2022-05-03 00:13:57 +05:00
chrislu 94635e9b5c filer: add filer group 2022-05-01 21:59:16 -07:00
Konstantin Lebedev 50c4f62ed4 ydb-go-sdk move to v3 2022-05-02 02:07:47 +05:00
chrislu 5d081d5c3c skip if already copied to remote 
fix https://github.com/chrislusf/seaweedfs/issues/2993
 2022-04-30 20:03:04 -07:00
Konstantin Lebedev 03aec9ce7e avoid panic if hashicorp raft not initialized 2022-04-30 13:56:35 +05:00
chrislu 94f824e1ce volume: sync to disk before copying volume files 
address https://github.com/chrislusf/seaweedfs/issues/2976
 2022-04-26 13:03:43 -07:00
guol-fnst 8b2552f944 fix http response error code 2022-04-24 11:07:27 +08:00
a 1d6a9e66b6 Merge branch 'master' into a 2022-04-20 14:01:42 -07:00
Konstantin Lebedev 1e35b4929f shell vacuum volume by collection and volume id 2022-04-18 18:40:58 +05:00
leyou240 89eb87c1d1
Merge branch 'master' into slices.SortFunc 2022-04-18 10:39:29 +08:00
justin 3551ca2fcf enhancement: replace sort.Slice with slices.SortFunc to reduce reflection 2022-04-18 10:35:43 +08:00
chrislu b4be56bb3b add timing info during ping operation 2022-04-16 12:45:49 -07:00
a 846858fb43 merge master 2022-04-12 14:42:29 -07:00
Chris Lu a87f57e47c
Merge pull request #2868 from kmlebedev/hashicorp_raft 
hashicorp raft
 2022-04-10 23:00:05 -07:00
Konstantin Lebedev d2fe0fae33 use the full ServerAddress as the identifier 2022-04-11 10:44:44 +05:00
Chris Lu 9f4ecd50ac
Merge pull request #2889 from blacktear23/filer-ui-bootstrap-style 
Change filer UI icon from picture to icon font.
 2022-04-08 00:50:32 -07:00
yulai.li ceb058ddd9 Change filer UI icon from picture to icon font. 2022-04-08 15:07:56 +08:00
chrislu 7f26676f57 adjust td width 2022-04-07 22:34:32 -07:00
yulai.li dff63e9024 Make operation buttons show when table row hovered 2022-04-08 13:10:12 +08:00
chrislu 4dba102137 use icon instead of button text 2022-04-07 11:15:01 -07:00
Chris Lu bbc95dac47
Merge pull request #2885 from binbinshi/master 
fix: master lose some volumes
 2022-04-07 10:16:19 -07:00
Konstantin Lebedev 104ea7b029 master ui add raft stats and Max Volume Id 2022-04-07 20:52:01 +05:00
a 41d396edc4 Merge branch 'master' into a 2022-04-07 08:22:59 -07:00
Konstantin Lebedev a2fdb3e277 fix master ui 2022-04-07 19:37:40 +05:00
Konstantin Lebedev 35bc67f030 fix conflicts 2022-04-07 18:52:13 +05:00
Konstantin Lebedev f5246b748d Merge branch 'new_master' into hashicorp_raft 
# Conflicts:
#	weed/pb/master_pb/master.pb.go
 2022-04-07 18:50:27 +05:00
yulai.li 11a9f993a8 Make filer-ui bootstrap style 2022-04-07 20:17:00 +08:00
Konstantin Lebedev 7ff248d5cd refactor OnPeerUpdate 2022-04-07 16:23:22 +05:00
Konstantin Lebedev 85d80fd36d fix removing old raft server 2022-04-07 15:31:37 +05:00
chrislu abe3cc6df2 filer UI touch up 2022-04-07 01:25:55 -07:00
yulai.li 2347c21cdd Tune filer UI add rename feature 2022-04-07 15:19:41 +08:00
yulai.li 2454020a92 Add upload progress support 2022-04-07 15:19:37 +08:00
yulai.li 4f87ee7755 Add create directory and delete web UI features for filer 2022-04-07 15:19:33 +08:00
shibinbin c20e1edd99 fix: master lose some volumes 2022-04-07 15:18:28 +08:00
eddy-gfx ec53eec94f
Merge branch 'master' into a 2022-04-06 18:45:55 -05:00
Konstantin Lebedev 14a2cc83bf raft update peers via OnPeerUpdate 2022-04-06 21:17:04 +05:00
Konstantin Lebedev 357aa818fe add raft shell cmds 2022-04-06 15:23:53 +05:00
chrislu 79b8e6a8c3 add a place holder for later merge consecutive chunks 2022-04-06 00:28:48 -07:00
Konstantin Lebedev c1aeded2dd force raft bootstrap 
avoid err bootstrap only works on new clusters
 2022-04-06 12:18:43 +05:00
chrislu bc888226fc erasure coding: tracking encoded/decoded volumes 
If an EC shard is created but not spread to other servers, the masterclient would think this shard is not located here.
 2022-04-05 19:03:02 -07:00
Konstantin Lebedev 0e796a5582 rm set NoSnapshotRestoreOnStart 2022-04-05 18:28:42 +05:00
Konstantin Lebedev 68f11b9687 sleep bootstraping 2022-04-05 15:42:19 +05:00
Konstantin Lebedev 1ae7f509f5 LocalID from raw string 2022-04-05 13:50:39 +05:00
Konstantin Lebedev 89d32a0670 fix glog Info 2022-04-05 13:43:34 +05:00
Konstantin Lebedev 17c6e8e39f Merge branch 'new_master' into hashicorp_raft 
# Conflicts:
#	go.mod
#	go.sum
 2022-04-05 13:29:46 +05:00
Konstantin Lebedev b7cdde14ae auto bootstraping and update peers 2022-04-05 13:17:53 +05:00
Konstantin Lebedev 622297f1a7 add stats raft handler 2022-04-04 19:16:06 +05:00
Konstantin Lebedev 1a1e5778c3 fix cluster status 2022-04-04 18:52:08 +05:00
Konstantin Lebedev 14dd971890 hashicorp raft with state machine 2022-04-04 17:51:51 +05:00
Konstantin Lebedev c514710b7b initial add hashicorp raft 2022-04-04 13:50:56 +05:00
chrislu 6d55716fc3 use constants 2022-04-01 16:55:26 -07:00
chrislu bbbbbd70a4 master supports grpc ping 2022-04-01 16:50:58 -07:00
chrislu 743ad690b6 filer supports grpc ping 2022-04-01 16:44:58 -07:00
chrislu 2305508b65 refactor: separate into two files 2022-04-01 16:40:49 -07:00
chrislu 800cbc004c volume server adds ping function 2022-04-01 16:37:06 -07:00
a 549e341c6f resolv 2022-03-30 06:31:19 -05:00
a bc603e534f diff 2022-03-28 16:50:28 +00:00
Konstantin Lebedev 84b7b83517 fix permission mkdir snapshot 
avoid open file operation not permitted
 2022-03-28 18:41:52 +05:00
Chris Lu 93615b2a49
Merge pull request #2828 from guo-sj/fix-tagging-error 
Fix delete all "Seaweed-" prefixed tagging error
 2022-03-28 00:35:22 -07:00
guosj dd2b9d93cc update code according to PR's comment 2022-03-28 15:23:31 +08:00
guosj 02076f01a6 fix tagging error 2022-03-28 14:50:48 +08:00
chrislu 21e0898631 refactor: change masters from a slice to a map 2022-03-26 13:33:17 -07:00
chrislu 4ba7127ab1 refactor 2022-03-26 13:13:19 -07:00
chrislu fba1cfc2d6 simplify a bit 2022-03-26 10:24:05 -07:00
chrislu a3411dd9da refactor 2022-03-26 10:21:26 -07:00
Chris Lu 1b27f96669
Merge pull request #2814 from kmlebedev/fix_remove_deleted_peers 
Fix remove deleted peers
 2022-03-25 11:36:15 -07:00
Konstantin Lebedev ddd3945c26 fix remove deleted peers of raft server 
https://github.com/chrislusf/seaweedfs/issues/2804
 2022-03-25 15:09:38 +05:00
Konstantin Lebedev c1450bf9fe always clear previous log to avoid server is promotable 
https://github.com/chrislusf/seaweedfs/issues/2804
 2022-03-25 13:40:19 +05:00
Chris Lu 89d84e275b
Merge pull request #2759 from kmlebedev/skip_wait_cancelled_request 
Need to exit waiting if request is was canceled
 2022-03-24 12:21:44 -07:00
Konstantin Lebedev 0b790d2bbf exclude the replication from the concurrentUploadLimitMB 2022-03-24 13:54:42 +05:00
chrislu 9b04f17555 remove dead code 2022-03-23 23:18:32 -07:00
chrislu 0b5faef612 fix 2022-03-23 23:16:54 -07:00
chrislu 0562fceb99 volume: fail fast if too many concurrent requests, to avoid dead lock due to replication. 
fix https://github.com/chrislusf/seaweedfs/issues/2755
 2022-03-23 22:53:58 -07:00
zzq09494 9f16df736e fix-filer: calculation error of the method skipCheckParentDirEntry 2022-03-19 09:15:42 +08:00
elee b25d03340b Merge branch 'a' of github.com:gfxlabs/seaweedfs into a 2022-03-17 04:51:59 -05:00
elee 921535001a arangodb adapter 2022-03-17 04:49:26 -05:00
chrislu 4042fdf3bb rename to skipCheckParentDir 
related to https://github.com/chrislusf/seaweedfs/pull/2761

It's better to default to false.
 2022-03-16 23:55:31 -07:00
zzq09494 40b0033fa7 go fmt 2022-03-17 14:19:48 +08:00
zzq09494 81cce4b4c3 filer: support uploading file without needEnsureParentDir 2022-03-17 10:53:47 +08:00
zzq09494 a6a8892255 Revert "filer: support uploading file without needEnsureParentDir" 
This reverts commit a93c4947ba.
 2022-03-17 10:27:17 +08:00
zzq09494 a93c4947ba filer: support uploading file without needEnsureParentDir 2022-03-17 10:18:23 +08:00
Konstantin Lebedev f43c6daeda Need to exit waiting if request is was canceled 2022-03-15 19:55:22 +05:00
banjiaojuhao f28dbbe5c5 [bugfix] filer: 1. Delete uploaded chunks when upload failed. 2. Report error when upload is interrupted by user. 2022-03-10 11:40:39 +08:00
chrislu 6d3db4445b buffer for all range requests 2022-03-07 01:56:47 -08:00
chrislu f3bcbeb60a a little optimization 2022-03-07 00:24:59 -08:00
Chris Lu 1e7fcef581
Merge pull request #2729 from banjiaojuhao/filer_metadata-resolve-manifest 
filer: support get metadata with resolved manifest chunk
 2022-03-07 00:05:46 -08:00
chrislu bb0b784544 minor 2022-03-07 00:04:59 -08:00
banjiaojuhao bfcc9ca808 filer: support metadata with resolved manifest chunk 2022-03-07 15:47:51 +08:00
banjiaojuhao 71f3046841 filer: add back isAppend function 2022-03-07 15:41:07 +08:00
Konstantin Lebedev cf444ebd07 Set default leveldb2 enabled 
avoid Filer store is enabled for both leveldb2 and mysql
 2022-03-06 18:27:25 +05:00
chrislu f1713c96ae avoid possible runtime error: index out of range [0] with length 0 2022-03-05 21:14:31 -08:00
chrislu a96d4254e9 filer, s3, volume server: a bit memory optimization 2022-03-02 20:15:28 -08:00
chrislu b2a148cb4c use file size as max range 2022-02-26 03:00:08 -08:00
chrislu 28b395bef4 better control for reader caching 2022-02-26 02:16:47 -08:00
chrislu e423548673 rename: pass along entry metadata 2022-02-25 02:53:37 -08:00
chrislu 320637dc7a use "mv.from" for moving files 2022-02-23 15:34:42 -08:00
banjiaojuhao 6ab09e9071 filer_http: support uploading file with offset 2022-02-22 00:15:00 +08:00
banjiaojuhao e6126cef62 filer_web: support moving entry 2022-02-20 23:56:23 +08:00
banjiaojuhao 4c30934cd9 filer: support get file entry 2022-02-18 22:52:26 +08:00
Konstantin Lebedev 9ea09cc41c healthz check to avoid drain pod with last replicas 2022-02-16 14:18:36 +05:00
garenchan bd032eabe7 [UPDATE] Make heartbeat interval and election timeout of masters configurable. 2022-02-14 21:09:07 +08:00
root 7f0c793083 fix preconditions according to https://tools.ietf.org/id/draft-ietf-httpbis-p4-conditional-26.html#preconditions 2022-02-08 10:13:19 +08:00
chrislu 433fde4b18 move error to a separate file 
This file contains metric names for all errors
The naming convention is ErrorSomeThing = "error.some.thing"
 2022-02-04 22:57:51 -08:00
Chris Lu a23fcb9a7c
Merge pull request #2634 from kmlebedev/errorMetrics 
error metrics for filer and store
 2022-02-04 22:35:13 -08:00